Zhong, Xin
2010-Dec-09 09:30 UTC
[PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
This problem is found in meego testing:
http://bugs.meego.com/show_bug.cgi?id=6672
A file in btrfs is mmaped and the mmaped buffer is passed to pwrite to write to
the same page
of the same file. In btrfs_file_aio_write(), the pages is locked by
prepare_pages(). So when
btrfs_copy_from_user() is called, page fault happens and the same page needs to
be locked again
in filemap_fault(). The fix is to move iov_iter_fault_in_readable() before
prepage_pages() to make page
fault happen before pages are locked. And also disable page fault in critical
region in
btrfs_copy_from_user().
Reviewed-by: Yan, Zheng<zheng.z.yan@intel.com>
Signed-off-by: Zhong, Xin <xin.zhong@intel.com>
---
fs/btrfs/file.c | 92 ++++++++++++++++++++++++++++++++++++-------------------
1 files changed, 60 insertions(+), 32 deletions(-)
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index c1faded..66836d8 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -48,30 +48,34 @@ static noinline int btrfs_copy_from_user(loff_t pos, int
num_pages,
struct page **prepared_pages,
struct iov_iter *i)
{
- size_t copied;
+ size_t copied = 0;
int pg = 0;
int offset = pos & (PAGE_CACHE_SIZE - 1);
+ int total_copied = 0;
while (write_bytes > 0) {
size_t count = min_t(size_t,
PAGE_CACHE_SIZE - offset, write_bytes);
struct page *page = prepared_pages[pg];
-again:
- if (unlikely(iov_iter_fault_in_readable(i, count)))
- return -EFAULT;
-
- /* Copy data from userspace to the current page */
- copied = iov_iter_copy_from_user(page, i, offset, count);
+ /*
+ * Copy data from userspace to the current page
+ *
+ * Disable pagefault to avoid recursive lock since
+ * the pages are already locked
+ */
+ pagefault_disable();
+ copied = iov_iter_copy_from_user_atomic(page, i, offset, count);
+ pagefault_enable();
/* Flush processor''s dcache for this page */
flush_dcache_page(page);
iov_iter_advance(i, copied);
write_bytes -= copied;
+ total_copied += copied;
+ /* Return to btrfs_file_aio_write to fault page */
if (unlikely(copied == 0)) {
- count = min_t(size_t, PAGE_CACHE_SIZE - offset,
- iov_iter_single_seg_count(i));
- goto again;
+ break;
}
if (unlikely(copied < PAGE_CACHE_SIZE - offset)) {
@@ -81,7 +85,7 @@ again:
offset = 0;
}
}
- return 0;
+ return total_copied;
}
/*
@@ -854,6 +858,8 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb,
unsigned long last_index;
int will_write;
int buffered = 0;
+ int copied = 0;
+ int dirty_pages = 0;
will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) ||
(file->f_flags & O_DIRECT));
@@ -970,7 +976,17 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb,
WARN_ON(num_pages > nrptrs);
memset(pages, 0, sizeof(struct page *) * nrptrs);
- ret = btrfs_delalloc_reserve_space(inode, write_bytes);
+ /*
+ * Fault pages before locking them in prepare_pages
+ * to avoid recursive lock
+ */
+ if (unlikely(iov_iter_fault_in_readable(&i, write_bytes))) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ ret = btrfs_delalloc_reserve_space(inode,
+ num_pages << PAGE_CACHE_SHIFT);
if (ret)
goto out;
@@ -978,37 +994,49 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb,
pos, first_index, last_index,
write_bytes);
if (ret) {
- btrfs_delalloc_release_space(inode, write_bytes);
+ btrfs_delalloc_release_space(inode,
+ num_pages << PAGE_CACHE_SHIFT);
goto out;
}
- ret = btrfs_copy_from_user(pos, num_pages,
+ copied = btrfs_copy_from_user(pos, num_pages,
write_bytes, pages, &i);
- if (ret == 0) {
+ dirty_pages = (copied + PAGE_CACHE_SIZE - 1) >>
+ PAGE_CACHE_SHIFT;
+
+ if (num_pages > dirty_pages) {
+ if (copied > 0)
+ atomic_inc(
+ &BTRFS_I(inode)->outstanding_extents);
+ btrfs_delalloc_release_space(inode,
+ (num_pages - dirty_pages) <<
+ PAGE_CACHE_SHIFT);
+ }
+
+ if (copied > 0) {
dirty_and_release_pages(NULL, root, file, pages,
- num_pages, pos, write_bytes);
+ dirty_pages, pos, copied);
}
btrfs_drop_pages(pages, num_pages);
- if (ret) {
- btrfs_delalloc_release_space(inode, write_bytes);
- goto out;
- }
- if (will_write) {
- filemap_fdatawrite_range(inode->i_mapping, pos,
- pos + write_bytes - 1);
- } else {
- balance_dirty_pages_ratelimited_nr(inode->i_mapping,
- num_pages);
- if (num_pages <
- (root->leafsize >> PAGE_CACHE_SHIFT) + 1)
- btrfs_btree_balance_dirty(root, 1);
- btrfs_throttle(root);
+ if (copied > 0) {
+ if (will_write) {
+ filemap_fdatawrite_range(inode->i_mapping, pos,
+ pos + copied - 1);
+ } else {
+ balance_dirty_pages_ratelimited_nr(
+ inode->i_mapping,
+ dirty_pages);
+ if (dirty_pages <
+ (root->leafsize >> PAGE_CACHE_SHIFT) + 1)
+ btrfs_btree_balance_dirty(root, 1);
+ btrfs_throttle(root);
+ }
}
- pos += write_bytes;
- num_written += write_bytes;
+ pos += copied;
+ num_written += copied;
cond_resched();
}
--
1.6.2.2
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Johannes Hirte
2011-Jan-27 13:09 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thursday 09 December 2010 10:30:14 Zhong, Xin wrote:> This problem is found in meego testing: > http://bugs.meego.com/show_bug.cgi?id=6672 > A file in btrfs is mmaped and the mmaped buffer is passed to pwrite to > write to the same page of the same file. In btrfs_file_aio_write(), the > pages is locked by prepare_pages(). So when btrfs_copy_from_user() is > called, page fault happens and the same page needs to be locked again in > filemap_fault(). The fix is to move iov_iter_fault_in_readable() before > prepage_pages() to make page fault happen before pages are locked. And > also disable page fault in critical region in btrfs_copy_from_user(). > > Reviewed-by: Yan, Zheng<zheng.z.yan@intel.com> > Signed-off-by: Zhong, Xin <xin.zhong@intel.com> > --- > fs/btrfs/file.c | 92 > ++++++++++++++++++++++++++++++++++++------------------- 1 files changed, > 60 insertions(+), 32 deletions(-) > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c > index c1faded..66836d8 100644 > --- a/fs/btrfs/file.c > +++ b/fs/btrfs/file.c > @@ -48,30 +48,34 @@ static noinline int btrfs_copy_from_user(loff_t pos, > int num_pages, struct page **prepared_pages, > struct iov_iter *i) > { > - size_t copied; > + size_t copied = 0; > int pg = 0; > int offset = pos & (PAGE_CACHE_SIZE - 1); > + int total_copied = 0; > > while (write_bytes > 0) { > size_t count = min_t(size_t, > PAGE_CACHE_SIZE - offset, write_bytes); > struct page *page = prepared_pages[pg]; > -again: > - if (unlikely(iov_iter_fault_in_readable(i, count))) > - return -EFAULT; > - > - /* Copy data from userspace to the current page */ > - copied = iov_iter_copy_from_user(page, i, offset, count); > + /* > + * Copy data from userspace to the current page > + * > + * Disable pagefault to avoid recursive lock since > + * the pages are already locked > + */ > + pagefault_disable(); > + copied = iov_iter_copy_from_user_atomic(page, i, offset, count); > + pagefault_enable(); > > /* Flush processor''s dcache for this page */ > flush_dcache_page(page); > iov_iter_advance(i, copied); > write_bytes -= copied; > + total_copied += copied; > > + /* Return to btrfs_file_aio_write to fault page */ > if (unlikely(copied == 0)) { > - count = min_t(size_t, PAGE_CACHE_SIZE - offset, > - iov_iter_single_seg_count(i)); > - goto again; > + break; > } > > if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { > @@ -81,7 +85,7 @@ again: > offset = 0; > } > } > - return 0; > + return total_copied; > } > > /* > @@ -854,6 +858,8 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > unsigned long last_index; > int will_write; > int buffered = 0; > + int copied = 0; > + int dirty_pages = 0; > > will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || > (file->f_flags & O_DIRECT)); > @@ -970,7 +976,17 @@ static ssize_t btrfs_file_aio_write(struct kiocb > *iocb, WARN_ON(num_pages > nrptrs); > memset(pages, 0, sizeof(struct page *) * nrptrs); > > - ret = btrfs_delalloc_reserve_space(inode, write_bytes); > + /* > + * Fault pages before locking them in prepare_pages > + * to avoid recursive lock > + */ > + if (unlikely(iov_iter_fault_in_readable(&i, write_bytes))) { > + ret = -EFAULT; > + goto out; > + } > + > + ret = btrfs_delalloc_reserve_space(inode, > + num_pages << PAGE_CACHE_SHIFT); > if (ret) > goto out; > > @@ -978,37 +994,49 @@ static ssize_t btrfs_file_aio_write(struct kiocb > *iocb, pos, first_index, last_index, > write_bytes); > if (ret) { > - btrfs_delalloc_release_space(inode, write_bytes); > + btrfs_delalloc_release_space(inode, > + num_pages << PAGE_CACHE_SHIFT); > goto out; > } > > - ret = btrfs_copy_from_user(pos, num_pages, > + copied = btrfs_copy_from_user(pos, num_pages, > write_bytes, pages, &i); > - if (ret == 0) { > + dirty_pages = (copied + PAGE_CACHE_SIZE - 1) >> > + PAGE_CACHE_SHIFT; > + > + if (num_pages > dirty_pages) { > + if (copied > 0) > + atomic_inc( > + &BTRFS_I(inode)->outstanding_extents); > + btrfs_delalloc_release_space(inode, > + (num_pages - dirty_pages) << > + PAGE_CACHE_SHIFT); > + } > + > + if (copied > 0) { > dirty_and_release_pages(NULL, root, file, pages, > - num_pages, pos, write_bytes); > + dirty_pages, pos, copied); > } > > btrfs_drop_pages(pages, num_pages); > - if (ret) { > - btrfs_delalloc_release_space(inode, write_bytes); > - goto out; > - } > > - if (will_write) { > - filemap_fdatawrite_range(inode->i_mapping, pos, > - pos + write_bytes - 1); > - } else { > - balance_dirty_pages_ratelimited_nr(inode->i_mapping, > - num_pages); > - if (num_pages < > - (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > - btrfs_btree_balance_dirty(root, 1); > - btrfs_throttle(root); > + if (copied > 0) { > + if (will_write) { > + filemap_fdatawrite_range(inode->i_mapping, pos, > + pos + copied - 1); > + } else { > + balance_dirty_pages_ratelimited_nr( > + inode->i_mapping, > + dirty_pages); > + if (dirty_pages < > + (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > + btrfs_btree_balance_dirty(root, 1); > + btrfs_throttle(root); > + } > } > > - pos += write_bytes; > - num_written += write_bytes; > + pos += copied; > + num_written += copied; > > cond_resched(); > }This patch breaks one of my Gentoo boxes. When I try to install/update via emerge, some packages hang. It seems that it''s always a "svn info" process that is stuck in kernel eating 100% CPU. I don''t know how svn is involved here, but reverting this patch makes the system work again. I''ll try to get a simple testcase. regards, Johannes -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Maria Wikström
2011-Jan-27 22:12 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
tor 2011-01-27 klockan 14:09 +0100 skrev Johannes Hirte:> On Thursday 09 December 2010 10:30:14 Zhong, Xin wrote: > > This problem is found in meego testing: > > http://bugs.meego.com/show_bug.cgi?id=6672 > > A file in btrfs is mmaped and the mmaped buffer is passed to pwrite to > > write to the same page of the same file. In btrfs_file_aio_write(), the > > pages is locked by prepare_pages(). So when btrfs_copy_from_user() is > > called, page fault happens and the same page needs to be locked again in > > filemap_fault(). The fix is to move iov_iter_fault_in_readable() before > > prepage_pages() to make page fault happen before pages are locked. And > > also disable page fault in critical region in btrfs_copy_from_user(). > > > > Reviewed-by: Yan, Zheng<zheng.z.yan@intel.com> > > Signed-off-by: Zhong, Xin <xin.zhong@intel.com> > > --- > > fs/btrfs/file.c | 92 > > ++++++++++++++++++++++++++++++++++++------------------- 1 files changed, > > 60 insertions(+), 32 deletions(-) > > > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c > > index c1faded..66836d8 100644 > > --- a/fs/btrfs/file.c > > +++ b/fs/btrfs/file.c > > @@ -48,30 +48,34 @@ static noinline int btrfs_copy_from_user(loff_t pos, > > int num_pages, struct page **prepared_pages, > > struct iov_iter *i) > > { > > - size_t copied; > > + size_t copied = 0; > > int pg = 0; > > int offset = pos & (PAGE_CACHE_SIZE - 1); > > + int total_copied = 0; > > > > while (write_bytes > 0) { > > size_t count = min_t(size_t, > > PAGE_CACHE_SIZE - offset, write_bytes); > > struct page *page = prepared_pages[pg]; > > -again: > > - if (unlikely(iov_iter_fault_in_readable(i, count))) > > - return -EFAULT; > > - > > - /* Copy data from userspace to the current page */ > > - copied = iov_iter_copy_from_user(page, i, offset, count); > > + /* > > + * Copy data from userspace to the current page > > + * > > + * Disable pagefault to avoid recursive lock since > > + * the pages are already locked > > + */ > > + pagefault_disable(); > > + copied = iov_iter_copy_from_user_atomic(page, i, offset, count); > > + pagefault_enable(); > > > > /* Flush processor''s dcache for this page */ > > flush_dcache_page(page); > > iov_iter_advance(i, copied); > > write_bytes -= copied; > > + total_copied += copied; > > > > + /* Return to btrfs_file_aio_write to fault page */ > > if (unlikely(copied == 0)) { > > - count = min_t(size_t, PAGE_CACHE_SIZE - offset, > > - iov_iter_single_seg_count(i)); > > - goto again; > > + break; > > } > > > > if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { > > @@ -81,7 +85,7 @@ again: > > offset = 0; > > } > > } > > - return 0; > > + return total_copied; > > } > > > > /* > > @@ -854,6 +858,8 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > > unsigned long last_index; > > int will_write; > > int buffered = 0; > > + int copied = 0; > > + int dirty_pages = 0; > > > > will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || > > (file->f_flags & O_DIRECT)); > > @@ -970,7 +976,17 @@ static ssize_t btrfs_file_aio_write(struct kiocb > > *iocb, WARN_ON(num_pages > nrptrs); > > memset(pages, 0, sizeof(struct page *) * nrptrs); > > > > - ret = btrfs_delalloc_reserve_space(inode, write_bytes); > > + /* > > + * Fault pages before locking them in prepare_pages > > + * to avoid recursive lock > > + */ > > + if (unlikely(iov_iter_fault_in_readable(&i, write_bytes))) { > > + ret = -EFAULT; > > + goto out; > > + } > > + > > + ret = btrfs_delalloc_reserve_space(inode, > > + num_pages << PAGE_CACHE_SHIFT); > > if (ret) > > goto out; > > > > @@ -978,37 +994,49 @@ static ssize_t btrfs_file_aio_write(struct kiocb > > *iocb, pos, first_index, last_index, > > write_bytes); > > if (ret) { > > - btrfs_delalloc_release_space(inode, write_bytes); > > + btrfs_delalloc_release_space(inode, > > + num_pages << PAGE_CACHE_SHIFT); > > goto out; > > } > > > > - ret = btrfs_copy_from_user(pos, num_pages, > > + copied = btrfs_copy_from_user(pos, num_pages, > > write_bytes, pages, &i); > > - if (ret == 0) { > > + dirty_pages = (copied + PAGE_CACHE_SIZE - 1) >> > > + PAGE_CACHE_SHIFT; > > + > > + if (num_pages > dirty_pages) { > > + if (copied > 0) > > + atomic_inc( > > + &BTRFS_I(inode)->outstanding_extents); > > + btrfs_delalloc_release_space(inode, > > + (num_pages - dirty_pages) << > > + PAGE_CACHE_SHIFT); > > + } > > + > > + if (copied > 0) { > > dirty_and_release_pages(NULL, root, file, pages, > > - num_pages, pos, write_bytes); > > + dirty_pages, pos, copied); > > } > > > > btrfs_drop_pages(pages, num_pages); > > - if (ret) { > > - btrfs_delalloc_release_space(inode, write_bytes); > > - goto out; > > - } > > > > - if (will_write) { > > - filemap_fdatawrite_range(inode->i_mapping, pos, > > - pos + write_bytes - 1); > > - } else { > > - balance_dirty_pages_ratelimited_nr(inode->i_mapping, > > - num_pages); > > - if (num_pages < > > - (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > > - btrfs_btree_balance_dirty(root, 1); > > - btrfs_throttle(root); > > + if (copied > 0) { > > + if (will_write) { > > + filemap_fdatawrite_range(inode->i_mapping, pos, > > + pos + copied - 1); > > + } else { > > + balance_dirty_pages_ratelimited_nr( > > + inode->i_mapping, > > + dirty_pages); > > + if (dirty_pages < > > + (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > > + btrfs_btree_balance_dirty(root, 1); > > + btrfs_throttle(root); > > + } > > } > > > > - pos += write_bytes; > > - num_written += write_bytes; > > + pos += copied; > > + num_written += copied; > > > > cond_resched(); > > } > > This patch breaks one of my Gentoo boxes. When I try to install/update via > emerge, some packages hang. It seems that it''s always a "svn info" process > that is stuck in kernel eating 100% CPU. I don''t know how svn is involved > here, but reverting this patch makes the system work again. I''ll try to get a > simple testcase. > > regards, > JohannesThe same thing happens here. The only way to kill the process (svn info) is to restart the computer. Running vanilla 2.6.37 without patches. #strace emerge libgcrypt -- cut -- open("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 7 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb713a000 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 _llseek(7, 416, [416], SEEK_SET) = 0 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 stat64("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 dup(1) = 8 fcntl64(8, F_GETFL) = 0x2 (flags O_RDWR) fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7005000 _llseek(8, 0, 0xbff80cfc, SEEK_CUR) = -1 ESPIPE (Illegal seek) fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 stat64("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/environment", {st_mode=S_IFREG|0664, st_size=105970, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID| SIGCHLD, child_tidptr=0xb7430728) = 11665 close(6) = 0 gettimeofday({1296162836, 126192}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 1 ([{fd=5, revents=POLLIN}]) read(5, ">>> Preparing source in /var/tmp"..., 4096) = 91 read(5, 0xb713b000, 4096) = -1 EAGAIN (Resource temporarily unavailable) write(8, ">>> Preparing source in /var/tmp"..., 91>>> Preparing source in /var/tmp/portage/dev-libs/libgcrypt-1.4.6/work/libgcrypt-1.4.6 ... ) = 91 write(7, ">>> Preparing source in /var/tmp"..., 91) = 91 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) The last line is repeated every 3sec. // Maria -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Zhong, Xin
2011-Jan-28 01:26 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Please try the fix in below link: http://www.spinics.net/lists/linux-btrfs/msg08051.html Thanks! -----Original Message----- From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-owner@vger.kernel.org] On Behalf Of Maria Wikstr?m Sent: Friday, January 28, 2011 6:12 AM To: johannes.hirte@fem.tu-ilmenau.de; Zhong, Xin Cc: linux-btrfs@vger.kernel.org Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page tor 2011-01-27 klockan 14:09 +0100 skrev Johannes Hirte:> On Thursday 09 December 2010 10:30:14 Zhong, Xin wrote: > > This problem is found in meego testing: > > http://bugs.meego.com/show_bug.cgi?id=6672 > > A file in btrfs is mmaped and the mmaped buffer is passed to pwrite to > > write to the same page of the same file. In btrfs_file_aio_write(), the > > pages is locked by prepare_pages(). So when btrfs_copy_from_user() is > > called, page fault happens and the same page needs to be locked again in > > filemap_fault(). The fix is to move iov_iter_fault_in_readable() before > > prepage_pages() to make page fault happen before pages are locked. And > > also disable page fault in critical region in btrfs_copy_from_user(). > > > > Reviewed-by: Yan, Zheng<zheng.z.yan@intel.com> > > Signed-off-by: Zhong, Xin <xin.zhong@intel.com> > > --- > > fs/btrfs/file.c | 92 > > ++++++++++++++++++++++++++++++++++++------------------- 1 files changed, > > 60 insertions(+), 32 deletions(-) > > > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c > > index c1faded..66836d8 100644 > > --- a/fs/btrfs/file.c > > +++ b/fs/btrfs/file.c > > @@ -48,30 +48,34 @@ static noinline int btrfs_copy_from_user(loff_t pos, > > int num_pages, struct page **prepared_pages, > > struct iov_iter *i) > > { > > - size_t copied; > > + size_t copied = 0; > > int pg = 0; > > int offset = pos & (PAGE_CACHE_SIZE - 1); > > + int total_copied = 0; > > > > while (write_bytes > 0) { > > size_t count = min_t(size_t, > > PAGE_CACHE_SIZE - offset, write_bytes); > > struct page *page = prepared_pages[pg]; > > -again: > > - if (unlikely(iov_iter_fault_in_readable(i, count))) > > - return -EFAULT; > > - > > - /* Copy data from userspace to the current page */ > > - copied = iov_iter_copy_from_user(page, i, offset, count); > > + /* > > + * Copy data from userspace to the current page > > + * > > + * Disable pagefault to avoid recursive lock since > > + * the pages are already locked > > + */ > > + pagefault_disable(); > > + copied = iov_iter_copy_from_user_atomic(page, i, offset, count); > > + pagefault_enable(); > > > > /* Flush processor's dcache for this page */ > > flush_dcache_page(page); > > iov_iter_advance(i, copied); > > write_bytes -= copied; > > + total_copied += copied; > > > > + /* Return to btrfs_file_aio_write to fault page */ > > if (unlikely(copied == 0)) { > > - count = min_t(size_t, PAGE_CACHE_SIZE - offset, > > - iov_iter_single_seg_count(i)); > > - goto again; > > + break; > > } > > > > if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { > > @@ -81,7 +85,7 @@ again: > > offset = 0; > > } > > } > > - return 0; > > + return total_copied; > > } > > > > /* > > @@ -854,6 +858,8 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > > unsigned long last_index; > > int will_write; > > int buffered = 0; > > + int copied = 0; > > + int dirty_pages = 0; > > > > will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || > > (file->f_flags & O_DIRECT)); > > @@ -970,7 +976,17 @@ static ssize_t btrfs_file_aio_write(struct kiocb > > *iocb, WARN_ON(num_pages > nrptrs); > > memset(pages, 0, sizeof(struct page *) * nrptrs); > > > > - ret = btrfs_delalloc_reserve_space(inode, write_bytes); > > + /* > > + * Fault pages before locking them in prepare_pages > > + * to avoid recursive lock > > + */ > > + if (unlikely(iov_iter_fault_in_readable(&i, write_bytes))) { > > + ret = -EFAULT; > > + goto out; > > + } > > + > > + ret = btrfs_delalloc_reserve_space(inode, > > + num_pages << PAGE_CACHE_SHIFT); > > if (ret) > > goto out; > > > > @@ -978,37 +994,49 @@ static ssize_t btrfs_file_aio_write(struct kiocb > > *iocb, pos, first_index, last_index, > > write_bytes); > > if (ret) { > > - btrfs_delalloc_release_space(inode, write_bytes); > > + btrfs_delalloc_release_space(inode, > > + num_pages << PAGE_CACHE_SHIFT); > > goto out; > > } > > > > - ret = btrfs_copy_from_user(pos, num_pages, > > + copied = btrfs_copy_from_user(pos, num_pages, > > write_bytes, pages, &i); > > - if (ret == 0) { > > + dirty_pages = (copied + PAGE_CACHE_SIZE - 1) >> > > + PAGE_CACHE_SHIFT; > > + > > + if (num_pages > dirty_pages) { > > + if (copied > 0) > > + atomic_inc( > > + &BTRFS_I(inode)->outstanding_extents); > > + btrfs_delalloc_release_space(inode, > > + (num_pages - dirty_pages) << > > + PAGE_CACHE_SHIFT); > > + } > > + > > + if (copied > 0) { > > dirty_and_release_pages(NULL, root, file, pages, > > - num_pages, pos, write_bytes); > > + dirty_pages, pos, copied); > > } > > > > btrfs_drop_pages(pages, num_pages); > > - if (ret) { > > - btrfs_delalloc_release_space(inode, write_bytes); > > - goto out; > > - } > > > > - if (will_write) { > > - filemap_fdatawrite_range(inode->i_mapping, pos, > > - pos + write_bytes - 1); > > - } else { > > - balance_dirty_pages_ratelimited_nr(inode->i_mapping, > > - num_pages); > > - if (num_pages < > > - (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > > - btrfs_btree_balance_dirty(root, 1); > > - btrfs_throttle(root); > > + if (copied > 0) { > > + if (will_write) { > > + filemap_fdatawrite_range(inode->i_mapping, pos, > > + pos + copied - 1); > > + } else { > > + balance_dirty_pages_ratelimited_nr( > > + inode->i_mapping, > > + dirty_pages); > > + if (dirty_pages < > > + (root->leafsize >> PAGE_CACHE_SHIFT) + 1) > > + btrfs_btree_balance_dirty(root, 1); > > + btrfs_throttle(root); > > + } > > } > > > > - pos += write_bytes; > > - num_written += write_bytes; > > + pos += copied; > > + num_written += copied; > > > > cond_resched(); > > } > > This patch breaks one of my Gentoo boxes. When I try to install/update via > emerge, some packages hang. It seems that it's always a "svn info" process > that is stuck in kernel eating 100% CPU. I don't know how svn is involved > here, but reverting this patch makes the system work again. I'll try to get a > simple testcase. > > regards, > JohannesThe same thing happens here. The only way to kill the process (svn info) is to restart the computer. Running vanilla 2.6.37 without patches. #strace emerge libgcrypt -- cut -- open("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 7 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb713a000 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 _llseek(7, 416, [416], SEEK_SET) = 0 fstat64(7, {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 stat64("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", {st_mode=S_IFREG|0660, st_size=416, ...}) = 0 dup(1) = 8 fcntl64(8, F_GETFL) = 0x2 (flags O_RDWR) fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7005000 _llseek(8, 0, 0xbff80cfc, SEEK_CUR) = -1 ESPIPE (Illegal seek) fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0 stat64("/var/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/environment", {st_mode=S_IFREG|0664, st_size=105970, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID| SIGCHLD, child_tidptr=0xb7430728) = 11665 close(6) = 0 gettimeofday({1296162836, 126192}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 1 ([{fd=5, revents=POLLIN}]) read(5, ">>> Preparing source in /var/tmp"..., 4096) = 91 read(5, 0xb713b000, 4096) = -1 EAGAIN (Resource temporarily unavailable) write(8, ">>> Preparing source in /var/tmp"..., 91>>> Preparing source in /var/tmp/portage/dev-libs/libgcrypt-1.4.6/work/libgcrypt-1.4.6 ... ) = 91 write(7, ">>> Preparing source in /var/tmp"..., 91) = 91 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) The last line is repeated every 3sec. // Maria -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Johannes Hirte
2011-Jan-28 02:54 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Friday 28 January 2011 02:26:43 Zhong, Xin wrote:> Please try the fix in below link: > http://www.spinics.net/lists/linux-btrfs/msg08051.html > > Thanks!This doesn''t fix it for me. At least there is a difference. Whereas the svn process started consuming 100% CPU without any further interaction before, the system just hang now. The svn process starts eating the CPU when I cancel the emerge via ctrl-c. Additional I see a flush-btrfs task now consuming CPU time. regards, Johannes -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Zhong, Xin
2011-Jan-28 03:53 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Could you describe the steps to recreate it? It will be a great help for me to look further. Thanks! -----Original Message----- From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] Sent: Friday, January 28, 2011 10:55 AM To: Zhong, Xin Cc: Maria Wikström; linux-btrfs@vger.kernel.org Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page On Friday 28 January 2011 02:26:43 Zhong, Xin wrote:> Please try the fix in below link: > http://www.spinics.net/lists/linux-btrfs/msg08051.html > > Thanks!This doesn't fix it for me. At least there is a difference. Whereas the svn process started consuming 100% CPU without any further interaction before, the system just hang now. The svn process starts eating the CPU when I cancel the emerge via ctrl-c. Additional I see a flush-btrfs task now consuming CPU time. regards, Johannes
Maria Wikström
2011-Jan-28 16:47 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
fre 2011-01-28 klockan 03:54 +0100 skrev Johannes Hirte:> On Friday 28 January 2011 02:26:43 Zhong, Xin wrote: > > Please try the fix in below link: > > http://www.spinics.net/lists/linux-btrfs/msg08051.html > > > > Thanks! > > This doesn''t fix it for me. At least there is a difference. Whereas the svn > process started consuming 100% CPU without any further interaction before, the > system just hang now. The svn process starts eating the CPU when I cancel the > emerge via ctrl-c. Additional I see a flush-btrfs task now consuming CPU time. > > regards, > Johannes > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >The patch makes the process exit cleanly but complains that there is no space left. It should be a few GB but it is only a few MB!? I delete 5GB and controls that the space is usable. Try again but this time it almost hangs the system, I can blink the leds and move the mouse but noting more. I boot 2.6.36.1 and use a snapshot as root, "emerge libgcrypt" compiles and installs fine. Boots back into 2.6.37 and try again and the system hangs again. // Maria -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Rui Miguel Silva
2011-Jan-28 18:27 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Friday, January 28, 2011 04:47:21 pm Maria Wikström wrote:> fre 2011-01-28 klockan 03:54 +0100 skrev Johannes Hirte: > > On Friday 28 January 2011 02:26:43 Zhong, Xin wrote: > > > Please try the fix in below link: > > > http://www.spinics.net/lists/linux-btrfs/msg08051.html > > > > > > Thanks! > > > > This doesn''t fix it for me. At least there is a difference. Whereas the > > svn process started consuming 100% CPU without any further interaction > > before, the system just hang now. The svn process starts eating the CPU > > when I cancel the emerge via ctrl-c. Additional I see a flush-btrfs task > > now consuming CPU time. > > > > regards, > > > > Johannes > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > The patch makes the process exit cleanly but complains that there is no > space left. It should be a few GB but it is only a few MB!? I delete 5GB > and controls that the space is usable. Try again but this time it almost > hangs the system, I can blink the leds and move the mouse but noting > more. > I boot 2.6.36.1 and use a snapshot as root, "emerge libgcrypt" compiles > and installs fine. Boots back into 2.6.37 and try again and the system > hangs again. > > // Maria >Hi Maria, I had something similar with vanilla 2.6.37 just for running: cscope -R -b -q allways hang complety the process, only solution reboot. I just applied the btrfs patches that arrived in the main tree between the 2.6.37 and 2.6.38-rc1 and they seem to fix the issue. I did not have the time to try to found which patch fixed the problem. Hope this could help you. Cheers, //Rui -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Maria Wikström
2011-Jan-29 15:38 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
fre 2011-01-28 klockan 18:27 +0000 skrev Rui Miguel Silva:> On Friday, January 28, 2011 04:47:21 pm Maria Wikström wrote: > > fre 2011-01-28 klockan 03:54 +0100 skrev Johannes Hirte: > > > On Friday 28 January 2011 02:26:43 Zhong, Xin wrote: > > > > Please try the fix in below link: > > > > http://www.spinics.net/lists/linux-btrfs/msg08051.html > > > > > > > > Thanks! > > > > > > This doesn''t fix it for me. At least there is a difference. Whereas the > > > svn process started consuming 100% CPU without any further interaction > > > before, the system just hang now. The svn process starts eating the CPU > > > when I cancel the emerge via ctrl-c. Additional I see a flush-btrfs task > > > now consuming CPU time. > > > > > > regards, > > > > > > Johannes > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > The patch makes the process exit cleanly but complains that there is no > > space left. It should be a few GB but it is only a few MB!? I delete 5GB > > and controls that the space is usable. Try again but this time it almost > > hangs the system, I can blink the leds and move the mouse but noting > > more. > > I boot 2.6.36.1 and use a snapshot as root, "emerge libgcrypt" compiles > > and installs fine. Boots back into 2.6.37 and try again and the system > > hangs again. > > > > // Maria > > > > Hi Maria, > I had something similar with vanilla 2.6.37 just for running: > cscope -R -b -q > > allways hang complety the process, only solution reboot. > I just applied the btrfs patches that arrived in the main tree between the > 2.6.37 and 2.6.38-rc1 and they seem to fix the issue. > > I did not have the time to try to found which patch fixed the problem. > > Hope this could help you. > > Cheers, > //RuiI have tried with both vanilla 2.6.38-rc2 and btrfs-unstable from git, both having the problem. Btrfs-unstable with the patch above behaves differently. The process can be killed with Ctrl + C but then the terminal and gnome-pty-helper eats cpu instead. They can be killed with SysRq + i together with everything else. // Maria -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Johannes Hirte
2011-Feb-01 23:34 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Friday 28 January 2011 04:53:24 Zhong, Xin wrote:> Could you describe the steps to recreate it? > It will be a great help for me to look further. Thanks!It''s a little strange. I have to systems with btrfs, both Gentoo-based. One is affected by this bug the other is not. On the affected system it is enough to do a ''emerge dev-libs/libgcrypt'' that should normaly compile and install libgcrypt. The emerge command is part of portage, the package management of Gentoo. The strace output looks similar to the one from Maria: open("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/.ipc_in", O_RDONLY| O_NONBLOCK|O_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFIFO|0770, st_size=0, ...}) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbff5f678) = -1 EINVAL (Invalid argument) open("/dev/ptmx", O_RDWR) = 5 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(5, TIOCGPTN, [2]) = 0 stat64("/dev/pts/2", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 getuid32() = 0 ioctl(5, TIOCSPTLCK, [0]) = 0 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(5, TIOCGPTN, [2]) = 0 stat64("/dev/pts/2", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 open("/dev/pts/2", O_RDWR|O_NOCTTY) = 6 ioctl(6, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(6, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(6, SNDCTL_TMR_START or SNDRV_TIMER_IOCTL_TREAD or TCSETS, {B38400 -opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 stat64("/root/.terminfo", 0xbff5e790) = -1 ENOENT (No such file or directory) stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=14, ...}) = 0 access("/etc/terminfo/x/xterm", R_OK) = 0 open("/etc/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 7 read(7, "\32\0010\0&\0\17\0\235\1l\5xterm|xterm terminal"..., 4097) = 3258 close(7) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TIOCGWINSZ, {ws_row=40, ws_col=207, ws_xpixel=0, ws_ypixel=0}) = 0 access("/usr/local/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/local/bin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/bin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/bin/stty", X_OK) = 0 stat64("/bin/stty", {st_mode=S_IFREG|0755, st_size=58836, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb753d728) = 2752 waitpid(2752, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 2752 --- SIGCHLD (Child exited) @ 0 (0) --- fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fstat64(5, {st_mode=S_IFCHR|0666, st_rdev=makedev(5, 2), ...}) = 0 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 -opost isig icanon echo ...}) = 0 open("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", O_WRONLY| O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 7 fstat64(7, {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 _llseek(7, 0, [480], SEEK_END) = 0 ioctl(7, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbff5fad8) = -1 ENOTTY (Inappropriate ioctl for device) fstat64(7, {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 _llseek(7, 0, [480], SEEK_CUR) = 0 stat64("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 dup(1) = 8 fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 ioctl(8, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 _llseek(8, 0, 0xbff5f820, SEEK_CUR) = -1 ESPIPE (Illegal seek) stat64("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/environment", {st_mode=S_IFREG|0664, st_size=106597, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb753d728) = 2753 close(6) = 0 gettimeofday({1296577457, 11117}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 1 ([{fd=5, revents=POLLIN}]) read(5, ">>> Preparing source in /home/tm"..., 4096) = 92 write(8, ">>> Preparing source in /home/tm"..., 92) = 92 write(7, ">>> Preparing source in /home/tm"..., 92) = 92 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000 <unfinished ...> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Zhong, Xin
2011-Feb-11 04:39 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Hi, Could you paste the output of sysrq+t here? Thanks! -----Original Message----- From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] Sent: Wednesday, February 02, 2011 7:35 AM To: Zhong, Xin Cc: Maria Wikström; linux-btrfs@vger.kernel.org Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page On Friday 28 January 2011 04:53:24 Zhong, Xin wrote:> Could you describe the steps to recreate it? > It will be a great help for me to look further. Thanks!It's a little strange. I have to systems with btrfs, both Gentoo-based. One is affected by this bug the other is not. On the affected system it is enough to do a 'emerge dev-libs/libgcrypt' that should normaly compile and install libgcrypt. The emerge command is part of portage, the package management of Gentoo. The strace output looks similar to the one from Maria: open("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/.ipc_in", O_RDONLY| O_NONBLOCK|O_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFIFO|0770, st_size=0, ...}) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbff5f678) = -1 EINVAL (Invalid argument) open("/dev/ptmx", O_RDWR) = 5 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(5, TIOCGPTN, [2]) = 0 stat64("/dev/pts/2", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 getuid32() = 0 ioctl(5, TIOCSPTLCK, [0]) = 0 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(5, TIOCGPTN, [2]) = 0 stat64("/dev/pts/2", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0 open("/dev/pts/2", O_RDWR|O_NOCTTY) = 6 ioctl(6, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(6, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(6, SNDCTL_TMR_START or SNDRV_TIMER_IOCTL_TREAD or TCSETS, {B38400 -opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 stat64("/root/.terminfo", 0xbff5e790) = -1 ENOENT (No such file or directory) stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=14, ...}) = 0 access("/etc/terminfo/x/xterm", R_OK) = 0 open("/etc/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 7 read(7, "\32\0010\0&\0\17\0\235\1l\5xterm|xterm terminal"..., 4097) = 3258 close(7) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, TIOCGWINSZ, {ws_row=40, ws_col=207, ws_xpixel=0, ws_ypixel=0}) = 0 access("/usr/local/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/local/bin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/usr/bin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/sbin/stty", X_OK) = -1 ENOENT (No such file or directory) access("/bin/stty", X_OK) = 0 stat64("/bin/stty", {st_mode=S_IFREG|0755, st_size=58836, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb753d728) = 2752 waitpid(2752, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 2752 --- SIGCHLD (Child exited) @ 0 (0) --- fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 fstat64(5, {st_mode=S_IFCHR|0666, st_rdev=makedev(5, 2), ...}) = 0 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 -opost isig icanon echo ...}) = 0 open("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", O_WRONLY| O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 7 fstat64(7, {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 _llseek(7, 0, [480], SEEK_END) = 0 ioctl(7, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbff5fad8) = -1 ENOTTY (Inappropriate ioctl for device) fstat64(7, {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 _llseek(7, 0, [480], SEEK_CUR) = 0 stat64("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/build.log", {st_mode=S_IFREG|0660, st_size=480, ...}) = 0 dup(1) = 8 fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 ioctl(8, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 fstat64(8, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 _llseek(8, 0, 0xbff5f820, SEEK_CUR) = -1 ESPIPE (Illegal seek) stat64("/home/tmp/portage/dev-libs/libgcrypt-1.4.6/temp/environment", {st_mode=S_IFREG|0664, st_size=106597, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb753d728) = 2753 close(6) = 0 gettimeofday({1296577457, 11117}, NULL) = 0 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 1 ([{fd=5, revents=POLLIN}]) read(5, ">>> Preparing source in /home/tm"..., 4096) = 92 write(8, ">>> Preparing source in /home/tm"..., 92) = 92 write(7, ">>> Preparing source in /home/tm"..., 92) = 92 poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000) = 0 (Timeout) poll([{fd=3, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=5, events=POLLIN| POLLERR|POLLHUP|POLLNVAL}], 2, 3000 <unfinished ...>
Maria Wikström
2011-Feb-18 11:31 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Seems like my reply got eaten by the lists spam filter, so I resend with attachment compressed. Should have thought of that :p fre 2011-02-11 klockan 12:39 +0800 skrev Zhong, Xin:> Hi, > > Could you paste the output of sysrq+t here? Thanks!Yes, it''s in the attachment. I tried latest btrfs from git (last commit Mon Feb 14 00:45:29 2011 +0000) but it hang so bad that I couldn''t get the output from sysrq+t to hit the disk. So the output is from vanilla 2.6.37 // Maria> -----Original Message----- > From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] > Sent: Wednesday, February 02, 2011 7:35 AM > To: Zhong, Xin > Cc: Maria Wikström; linux-btrfs@vger.kernel.org > Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page > > On Friday 28 January 2011 04:53:24 Zhong, Xin wrote: > > Could you describe the steps to recreate it? > > It will be a great help for me to look further. Thanks! > > It''s a little strange. I have to systems with btrfs, both Gentoo-based. One is > affected by this bug the other is not. On the affected system it is enough to do > a ''emerge dev-libs/libgcrypt'' that should normaly compile and install > libgcrypt. The emerge command is part of portage, the package management of > Gentoo. > The strace output looks similar to the one from Maria: >--=-9DLhnUr6//hYyzbOOvWY Content-Type: application/x-compressed-tar; name="vanilla_2.6.37_sysrq-t.txt.tar.gz" Content-Disposition: attachment; filename="vanilla_2.6.37_sysrq-t.txt.tar.gz" Content-Transfer-Encoding: base64 H4sIAAhVXk0AA+29W5MjK5Yu2M/zK+K9bFdxdYeyseqHYzbPY6fn7dgxGc4lUycjpChJkbmzf/2w AJfckRQBvqXMiExkbb0pMhxf4PDxsViXr2qzfnxUK/L37u+0X+2/73f//uPw98Ofh/+42Q8h1CH0 4P+Lez77ry/hjvaMPGDMSU8RR4g8IEz7jv/HA7qdCNd/L/uD2nlR/mo7KPwejv/9IL//xw4PuH8g +J9U/JN0D09qvXE79WQfvtjdxj7+8+G/vu//578f/H8/b789/NdBHez/9fZDB7X/8pD9/t//8eDH WkP989o8OHX4bHcFba0368Oplf960H7KDFaj8D/j/8cP6X+gP1H6FTTsOONUOfYQnxDkwbGBD47r 0zvgbwR80HkNVnosiII35Q+LaY1APBTGd4dlYTgNfyM0OwlR8Kax+6fm4AVE9+hUCH9AoLswAJay B+1Lktr0T4iV9Ol/qMfHh/9vp7T9Z8Ff/6//W2PEsELoX//74T8fPu8O6ye7W8HiO6x2avPJrjb7 v6E/MfqH/3+kokVsL7Ro/3xe72xoUPsGCStskBEnifYN7vVna14e7Sq2u30ZpdSPW/3FtyuZb3cY ygXFPZ0J+k19sS/PvqXQY10hIDevCOgbVL7B0vaQQgOVvr3n7ePj6thoatI3RoVvjbni5jBh0JzZ rvb20Wpogmn4Bhz1FTINKgzWagVyfVPrQxooxcsb0dyFRmITX2xqoitZR2GsKaK8i3Ls1LfV/nm9 Wb1s0gTAMAFw6cDggft9LrR1sPtxMg2hXxjTMOlL5cLSL2Qc2hoOO7dffbbK+Bn1yW7sTh3W2w00 KmHely4kaJKT0ORXu1u776tntbObw+rgBd2vjW+QiCBk8dLExpk0eKsvT+p5pQ7bp7X2LVlo6Gd8 hFu2haT/pnHElDEwXIe1eoTZDs0wWdwM6QhNw7R/VMPK7SzMVN4BvpQuYzww4XhoBp73+HeAr7eH 7RoWMTRGK2RSLi6dL0/2aaWVh4WjXLCEeCnu3VSum65HaaTuJ2tobw+wHFeft1tozakwSUv3DYJl x6Jozx6JjYV5TqCNvqKJ3sZVvXl5GuwO+sRxWMW09Mt1TNshtKH99rza6RffivFtFE9rgrWNH2xj rVn5TRT2hOP4lpC/G85rpKhEQ2rGrB79xwl7piW+FfujWyEsDa55DnsjhpG1JWTpNs9jSeHT6O3O wknttMNi7ODjyGI6IjUzJojytDkkYcImVLy9St0RESe8J/Kr1AaHNoo7JI1Cdlw0G//PsGigCVI+ zaxkcbq+7P0WGGRRgXTAt+XlaC4oi9NVP6829luAqI7BJIFxccXjwngn4j7/BUiU35T9JgpEVANw alX+sT1/B543/c4d7MGiFFE8ye+0jm141PXD41kZ4AJ8KVJKD74cPu88vzCvHL/I0uNXx50W46Gj Ox1e5uclhvMaPuixUH/88s9MaoTDtsPju6kJNcohPqlJQhT3qRenrpwX/Ifxh65OPbieaGONPv+b Oxy/xo5GFNx8RX5X4Wj3BLuwBEbmd46H/7X5ujZr9b+LW4SBmrXYhxZZIJ9+eKsaHAcmrZ4w7UbO Xnx28EdcGs+YsXoV2/Ec+fE57Kmohn16gRgDoBxXgW+g11W7hu+SwdMujX0qbwL6BB/uSo+60h59 2W/dYb37t992ri1l+pBWdPVS7p1CZ0uZZGtPkHwpS63HQv1S9s+camjHCOOanoSAGorppGYUouZN 5yvYOTf0CLuxhp46F1QqjuhU6EqweoEmhUiC4i5jrFMvj4egTli5l41O57/BTwpTTvp4In2r2NDL 80pvn55CSxSIHy+erLcikBQxB5N+97JZHScuEEjYuTAtX3+IqThUeUuosiFLw256QqaOVEHTbcDt JkDwbbvz5X+gf14FArYUCHrs5BkQiBwIPEhnNcbosVAPBP6ZU42RtCc9l+PyFKEGO8JPNaMQRW+C g2ffp84F7bCf5QM3l6AhLPueQQH1JCl4iRL32NNHyS4eiVd+lvvFd/CHBVh8MDcILl/EA4/nAw8D z482ntZhAdPyBay4jZT6ebfVdr9fbTd2BTMPaH44sRBSOvUZGZiJHfUk1vlGVmvzOKr0itWniA66 g47FBbA6rkM/aYP6oLh3NxmhQSWdYC4OqhPmF4Kmp/WnqMO8zlG6xdDEHUZTMiARd4jNscSfJzNo kv6UMhaqoQmeuXTbA+8+sgsjbZTPanYUohCaInAypbSM/MO3YlDEKNOb05vSP4mHUZOWaooUEEs4 Crw+bryRA6zgVhBWCSi1utIT9a0VwO8VLrm04XSmn19WXozn5xk8BbUZLta7eoKIuqjCPS6o1djy iC+lughGHFKxe+MFEZzxXOCFxSrS34Cz+oEiCqctz9qn58NqMmAUtH2sVJULU1WgkqnqfsJU1Rol ZdmFyYrqpuovuXnha5tXv3jz6rPNS1zavESXb14K6bFQv3n5Z6Y1x51ETDcvFfeW/rh5iaIr1snm JRGGXTe2Mt+8xGnz6s82r75tXm3zaptX27za5nU7pRC+rhQSizcvOVMKQb8ZVzjbbcA4bV5jsR4L 9ZuXf+ZUo6zhcFl4NHELNWKQ+FQzClHcp6QL0oRfvfEhobt8kMSKqBRiLG6ld1MKxbENcyqsuGhv Nqh9uF+F/YsV33+EvqV98NH6JjxMbfbbR7va26ew1+hw81y8hG+qFSKdI5QlQIhiwf3moML2Gi5Z iu2VbqYZalqd126erhJjuRhbzKWbpwxbPJHNsEWRdEvjC9XYAs+cagQfPEntJkJAjRG8m988gRAL bp5wfvNE6PzmKXQuGvO68eaJtpundvP0DoHgePN0DQgwWgwEroBkKH/YzGo6qsdCPRD4Zy6qd0Ea Gk07iHb0VDMKUdynRSSDozvfPDWS8VNJRrt+ug8++TPh3h7djS7iE16KT8KdLMMiPjHipZwBijxz NpKjs5Fc4GwkrzgbzQhFp4VK7yb6KERRn05wNz4sMhVh7DdjAcNYHzo36v3K37TE2QjG9qT3iY47 oEYKtq5hhlcgFEgeGrObf7/YFxvUgSun1rvRx6grNq//nRQ2hKvYVbCvDya6m7WGhlwgU7jYsp0O qIPOwmi9zKBT4rA5dL+AGpUqa9BR6/xyBsnFnfyVIPlLfOQ1SCZLIVnTEkjODZDlaIAsFxggy8wA +Q1IZlQfhaiG5PjwBUj2/b4OyWVvapDcILlB8u8JyRt72OwfXmXJi420LH4bkvmAdHaKH0inx0L1 KR6euXDPfYLk0VIsvtvqoxBFfToZYI0Py8wKLPb7GiSXvqlBcoPkBsm/JyTvv2/0av/in9pfhmRC lipWOadOielVBtSQuekR3LZmLNnSpE7whWqWDM9cVKzCu3m8xiWc0AfXI6cUOglR3KeoWB0Ioi7V 9AGAreiYStcoHpJPL+h73+tQQIZieydIDgJFNIBvata7w/fV8OJcmBBVztyhJxGQ/zyQ1WmO+IYk tKSK4dg3JXXWlAO/VgZTlBXD8S1DMPRGkuTDb9aT3u3TKmTF5j2+JQ4tnbdD6xr6hRDFj8Uf6Y7v GqIsPXdz3jt85vhLaMbKhM1JnhJ6LNSTPP/MxbhLEmEtNZy7jR85frp2HYUoepMXf1AqUUWPtdJw KodexO6Cm+/RqjH2NgRggn9y7HhXdac7W5Bsxsvs5rA+fIcVh+G6FpdHmLiRYeQNr4/uRVUo65RN V9NPWxNF9M1Y8JYucgRJ/JMy6OiF2EqiBwguxxYmkJvdap2aCrGRitHOMowT2rns6ociaIqyUgu4 90wUe4sxPu4PeU/r+vkrIfuXAT6PuX58J2Sx+6ct0agamZmpSzNEM3UoVHNFeOZVjepoKx/fLfRR iFqNanr4gkbVvqJRLXxTO76343s7vv+ex/cvSkNQ1NcgmS8m20UaVe1ysm2lHgv1ZNs/U65RNcnM G4So1qjGh881qtDv6xrVsjc1SG6Q3CD5N4bk1WZ7WLvvVyB56SUX72QJSzZ9zpId1mOhniU7XG53 YIzURyGqWXJ8+JwlQ7+vs+SyNzVIbpDcIPl3huTP28Pz48uny5C81L/e/0ogmfMckjunx0I9JHcX g8NcgWTe66MQ1ZAcH74Ayb7f1yG57E0NkhskN0j+PSFZHdRq79x1xQWlSz07Oe7fhGQueaZL9jVR lxwKlZAcnylUXMC7hT4KUae4OD58QXHh+30Fkovf1CC5QXKD5N8Tkr98fhmOquSLkMwWKy5EsG2Y moIJblVu+CXPagamx0K9KZh/5lTDidIdRpOo6b6mF8zMo7AGIYreRDBXbgCjLglku6CAUr+DxpwI bIu+cT0kR8ninNoO/8fqwwqSof0tJU4pUpeEdoKI/4pO+189nHgI/bTeH2KOkYArpXYbxA4kWae9 7IeV9lP0YFf2eeUbBnMpEQwRSv1h72a6YcUw+v9ayGS0OmxXKYtUSKnCi51/LXUOlrVfUydsGQS4 62ryg4NQUdulZDrqBQbmafvVnoVfgA9ASk3ofPdYH7fEWQdRVfd+KfD0W+56+4ptBGWL+SwbXK+n fJYiSzo8J6DM0IzPMpu4ny9U81lmp6GxicNMaDoBT6jpmZiBZxSiDDwF7YWZPex7SWdx+qAmqE0Y 7obsTfBPJXEw6sEzjm1cLYcvg4GYApuULkeG6WBKuVTsZMTP3for5JnZbQe7imA6cjNXGvTkpkEB w5DGlfPoIX21sX9CF3vIFFMUZCJ2kLguIsvwsgdLqZVV+jN0EBDT1KzE94/pFPGBAtqEpX4CPdKB PR4RxcFrbo/HXjRhaOL/M+FQrWi/DCI/nbjsFUTmS63V/GAXKH0lYhmdlTi6gEGhms7CM8XWahJ1 +ihErdI3PXxB6ev7fVXpW/impmFoGoamYfhNNQzHUKFXonjRbrGzGaLnUbyGQc1ZLWUmI8m06/VY qCbJ8MwFpa/jnYHJGcPuYeOSfOwkRMGbYlcwSg9zwF2ijO5leEHC/PFNUKBCpD8ObslEWS5Kptmi KF5jquWpi0MHTiF9MT0OnUlMO9wGrCIgPwOBSd4hrsaNKvZ41uKkrQEcMGz5urtpMDBlmRbXJEN1 grVgYO8c5vbf1LNB15knk4uZJxkcmugCcC81oR2b4xLOmKevicwzFKphDmfMM8Hc6I0VzQ1iOH8v nw2xBKMQRW86psmCh01qt2dy9HfrJu3Cmywe3xSd4/wf85IDzk8NVng/bzM/UqHZw+47HG0NzNwg JzikwiEXYK/Gwa5PoHDWEvDQ8iy9PdV9XI//HYBTHezuSe2+rEKibAxJroucrmNjUg3APuPSgsWI gt9geV7521H1XzwvwO21E/7rsSElNx6/H6r7er/Q7uD20TZ4BYvh0u7gsXPZ7tAj5pyYXrNBDcR2 nvlYnBmj8dEYjS8wRuO5MdrR8gGH/wPMthTjJE0kwVGIojehjjnwjw4PRwUxsQOmqSYGY/B/Yz0t 7jqB+xhoAf4J/jgmlL/P7hAk+9cYceHzbrtZ/7ddjbgSAgAUJ3wJfYo8Sq0PQZWaSFlccPG6rRhf woBcFS2y6vKc7TCECV32j2pYuZ0F1Av5WYZiYBmkddExep1WALQTlgFoEYISoVwk5eJ14pcn+7TS yvfsKBaMfHEqjvevd0aDcAi2nBly+I1+f9htwRnfhTDHNcmpb47wg7DJZfqakKhWxl8G8NV6+7oi mi/VevQo89GjYNEgO5nx9yw1pa+JqSlDof44wK8A/iVTN8ySfgWXpaacmbqlh89M3UK/r5q6Fb5p SUCMpjtuuuOmO/4pKPqn26+edi9xq7+IonKp7rhzc4NhJbDkVHQTkOtR180DlUFNCFQWC3Uomp4p u84L71b6KETVdd7x4QvXedDvyyha/qaGog1FG4p+JBR93H56xUzN06elXDQziqADcYJ2JCOPIne7 wCq5XeDqTLPxmXIuKpLbBS7LNDvnouKy20Xo93UuWvamhqINRRuKfiQUNeqgzCsoujDEpUcTl3HR nlnBeUYeucy5aE/1WKjnov3FbGRXuGhIYpGEqOaiMQPGGRcN/b7ORcve1FC0oWhD0Y+Eonq7+Wp3 B3MNRelSMwmdc1Ex4D53mcBZ4mhfExNHh0I9F7W4gouG0MFJiGouGh++4AKsX+OiZW9qKNpQtKHo h0FRvfv+fDheMF1E0cXmBH3GRRV1RGs9J4+9tBkX7Qehx0I1F4VnirloH3QHSYhaLhofvsRF+1e4 aOGbGoo2FG0o+mFQdK/365X9vELXUBROyAtRVEpIEDM1ypLS5lFzz2If8DH2AV8Q+4DPYx/MPBP8 HGLJM8GRWewDXhj7IPaJjM66YLsbasIrYY76Fo7tTl7gXy64GP+4xJapHkXvuPLsMLewhKUH853+ ZFMjIrTs5GifBRN5fbA7iK8QHY33o8VRsREwEc4Eq/64LHa77W71WW3MY1hUHNxdKS22Dbqhceo7 xlUYMzwaBl8YNVQ3aL8gtuKr2MoW3zlJd4atLsdWOWReX76mSzERh2qvr/jMFWztJtiaxZUZyry+ Yp9ybHVvYivjHUvY6hq2Nmxt2PpbYSu5iq18cbI1nWEr1JzzVjPkNWCTnwr1vDVa7Z8lW/OwduKt CNmMt3ohivs0w9ZQU8BbNRfjHzdsbdjasPU3wlZ6HVsX5tbpET7jrfictyqRx0NUKsVD9IVq3grP XMbWzl7F1iBEcZ8y3oqLeGvSCeCGrQ1bG7b+XtjKrmJrv/Duv0e8z7EVkuXmLq95khw+JsnhC5Lk 8CtJcsARtbvOW0uT5IQ+Zdjqa97mrRZFbIU/btjasLVh62+Erfw6ti6M490jQXNsFfSMt0Jfshqi 9Fio5q3wzGV9q1NHbKWCz3krCFHcpwxbfU0Bb9URW+GPG7Y2bG3Y+htha3cdWxdbW/EzOwF+bieg dK5vVTbpWyH4XS1vhWcu8VbO9Qlb2dDNoQ+EKO5TzltL7ARsusvizU6gYWvD1t8LW/ur2CoWe1X5 A3qOrWaOrT3qAvzMa0a/e1+otmSFZy7x1h7Rk06A234CfUmI4j7l2GoKeGs/6gRMw9aGrQ1bfwts HSNvv/yzv4ati3N72WEWefsINvOL/bnHKtQEj9VYqLcTyDxWjyFpKTWYuSiEDWGw0eA/gj0KUfCm CR5jbDsoUEHZAHvBsTD+07S7wg0xKLfWqmj0FqSnCf2Lcyq4fphV+M/fIM1HiNlmioNqhoEJTQ3r 7cqLstUrX9rbkCoMEEaW55i6ZbRsSrXVlztZ2cUWK/vDAJO4BkyLU3NbMwMm1xPE+iwnCz9TVvJR WckXKCv5XFk5WOF6IfFJCKjpNBNTu6ooRA0wTQqGIos8oaO+e8y/2zHlnCUiDUCAQL+eFDIwAHAc vxcwwTtma3a99TNBw1TQ5dGHb4wjkvXnMqE6kRqGfBQMkdcwZKHzjl9C7hxDaO6qoznOa7pBj4V6 DPHPTGoIJmpwY7I8PMSanvJJTRJiIYb4mQn26o45LrrgUQlv4CwNwAxD7ktuGoY0DPmpyidxVfkk FzsA6iFXPukhU+z7gwnDeU2MlRMK1cEo4JnLin13VD511s4cAKMQxX3KlE++pkCxn4z94I+b8qkp n5ry6TfCVnkNWwVa7ABocI6tvobkanxN85oxnIMv1Cv2zTQ3cke4dP4YNgohQo31WHqqGYUo7hOB IBt+3Qydnl0P+H+iIRGc8Kc+nGpYrNE94HqPUSfuxM+iQHFOpcTyzy9HZREtXcRR+EjQXg6TdMj/ qEgBEft7Wm4pQ32Uh7iw2CrwMxGi6/ipq/BT4g7Lk3C7l83q83Z/iAnuoL0QzoOUhlpoeNzw+Ibn 5ef90/Zlb81VPO4W47Hls5BBg+sI8bNtDqBZ4DWoCYHXYqEaj/PAa6+GDIqx05IQRTqAiZ9MCrx2 IWSQ7/fVkEEt8FoLGfSRkbSFDLqIovvDbv1srwcBFv1C90B/Rp+n9RGK94INWSj1LgsZ5GtiyKBQ qNYYdJdDBk1QdBK+spNaH4Uo6tMkfGV6+EL4SvdKWp/CNzUUbSjaUPTDoOjLfvi8Dsvk5eliWh+J Ft7/9n7+zVBUD5Qiy6ZclDPp5uEroSaEr4yFamcVVxy+Ev44RqCMQlRx0dPD51wU+n0FRYvf1FC0 oWhD0Q+Dou7xZf/5D/zPa+ErQ0L4hSiKnBNzDWsvcDfPKSwHpPMQa6TTY6E+xBrpJijKXc88Cowa VmRDjRFGn2pGIUr65NxgBab5FRVmqbu9OPXbfySlBSUzLawfgHslRwsvi3ezw8psV99264MdlA45 05GACVG+3pSWBNbbYNanhk7rxQZ8IcV5gL1sLF6BXWkQ1bX3Cy3AL+YprMFrh0GMusU+t1kuA0d7 zDjp5zowNjdBgZou5WlhtSYo6ZlXaYxzfuAETe9OuWdYmQnKTKUWH76kUruay6D8TY3GNBrTaMzH QdGjgdVVFF14GGSqG2YoisVgtDAiO73hnMY4mmiML9QfBml32UsBYcrBJyGIxXW0oxGDOgpRbMjH EygTSSaatPCz+vSCybv9m1A/1LxpAYqG/kU0eHoecc/D1Wq9Odjd7iXY8odlh0tv4OIIHZfMhfaI giVYmmI+jkNcxr6Dq923vX3ylOvbZhWmr0f8x4DVFCz9GuI3xG+Ify/Ef3iFNy81uySiz3mzh2M0 zIku1bkXMLXpgtgXqnkzPFN8FU1NuvQGIWp5c3z4Am+Gfl/lzYVvary5oWhD0Y+Dom9qHwRbqn0Q c96sOtZzLVQWokuSPOKX0nos1Ef88s+8ehUN2oeBsvTuFBMXhKi7ij4+fCmr93D9EqXwTQ1FG4o2 FP04KPqm9kEsjfdN9JyLdhgjqQ3L7BhFl5tFjvkNfaHeLNI/c4GLnrQPQayZ9iEJUax9oGqqfRC5 9mF8weTdo/ah/E1N+9C0Dw3xG+L/FO3D0qg4HvlkzptlbzXJiK6hOW+2I8e0lY5J6ZlCE054N9dH Iap5s+GXeTP0+zpvLntT480NRRuKfhgUHQ47t/8jhhL44xKKerK7GEUHR+fhN/DAcpZM5/kdoSbk d4yFeh3uLL+jUZa6gR8toHio6TW3p5pRiGLeDK6bTAoiTv7xY4wg5AeJMmTJuHsQQdJIIKDq/h+5 Kpn0C1A0NB3mVFhth/WTn+GD2sNKIbCKWQW9vcsipmwMWbZaPW1NFBFMmaxvxpV86oTMlEFHj0Qe 2tkG11EBzpSqPO4ZE8j9axpr5NQULEdV2BBEFxCwnlNIjsftNsYXCABlKvaum/l2epFE1/1rEick CYXqRPrloO6T3cCAXIa6pZ6TRJsZ1B2J05zh5ZGG5BhpSC6INCSzSENHGsckCt5FCMdj/tzyvDDS 0LGV6eWUNtbqadwQ6DcejRhEl2ruDXX3w6eGoA1BG4K+gaD7l+FpfbhCFpcG+SbazcmihzMiRIaX QudH7uPxVNTGAknPXD1yR7LIsGX9hCwmIcqu34aQoeF0godeRgQlSCjKU41jcwR1d0fQ+PpoDf74 xSPT8+PLJz8T4Khdfp5qcNngssHlW3Bp7CNEZ74Ml0vzJBKRE05ECDbd/DDsl3l2tiaE6LFQfbaG Z041lglpu2N2GcZCjRksPdWMQlSdrcMzEA3p5G05sY8y1ukZXIr7E844tlH99PyitD6s9Ge1+wSQ FGKzFeXuaaDZQLOBZhlouvWfL89/oMugudQcioicY4qcYwL2zJN0QU1I0hUL9aCZJemac8wQy9Mz QMsz0CxL0jUBTd8568QVq9XQyxlo3p9jNqhrUNeg7m2osxuz3l6DusU2S27ODx11sjvL7Gowy2uI 1WOh+jgNz1wKWxykQYm1WTEkafBJiJI+UddTTdgE8yjvnIQXhIJFp15O717cD+CHDeo+OtQ1eHoV np4uw9PS6D7Eyvxq2Eo7ZCaVnPOMifl1rsdCNRODZy7dl4R3m8jEUK9Iki8Zb4IQ5Uzs/EbY4xQP QX3zf0pMLL6pwVODpwZPi+Hp22V4Wux9aM8sV+wwhyeIWKNdHjYneB/GQn3YnNe8D+EyYvDzm8rs MmIo8wn0M0hRx8kleGK86/BwDk8kjcS94SlKFi8j4JvG4DTP6pNd+a+7Wm9Xn7fbL6OVVunCa6DX QO9XBT23s3b/rC6D3tIU9sSWXClkcWuhJsStjYVqTjaPWxuuFBRW2ZUC7slcO1YYTXaiHYNnzJUr BXt2pWDblUIDuqZn+7VAUz9atbG7i6BJFts4u8GJCWgej4FzlDP5lULn0pWCL9SDppteKfSUD9xD 9ylhTaiBIIzHmlGIkj55UmnP5X29BzMmy0VZIrJlhn+8u4hPRwePUp9ALIfB4AnrDPNjtX00q/1G Pe8/bw+Q6qUDAC1OFYilRDgiXZpuq/nkx8U+hr4pTKCp84YwrmvqHTtl3HDAfj3EOuzUZq/0ZcRa bKpsnWP6DLEy3zY592aGmjHctaz1Zk7PZIglMZojFnV0ilhJiHKaNw2hdnKVm0ROC6aCs7Oti2kQ m+rt1yRnUjgHPUvr6LDebqYoOoSUZaYi89ZtPAvv5Q14CzQWlsahvzxmqG7EfiFEfjH26+hrfNHb mJBky4dro1tKY4ZjGkSRzFLO4ksgcZYYMWRiiYX6NIjylSg9RMFliKVganOqSULUXIY4v5T7QaDU SzE1SxlrYnf7s38qunZZgMienPYm4d3OPm2/2tXn3Qh6HURs6H5ySlgvYs9FaDZJtjrsvq8O25VW G20fQUUADfLi+BfECT1MwTm26zHVy735ZD35jRyaQv+H0v4jzwn4mLv2efv4+E2tAaUlIJcqDSV+ S1bP/DQe4tC9bNZ/rp6tH7xPNoT4gGTdpGLIZIDTa0PmW4SYIcXRPRQaKAwVDNPqwiZJITw8K4ZW hbmBfprtav99HwYfGuFR8VDs+E3I0EeA9RL5WTv6uoMSIuRIVoEF1PQyBVqZTQhUMx/CrEoEIDTx xY6wX3wWaE1MlkSPSfwm+y9Ru6T23zca1hYsCUrKG+oZP62tcE3jV6sKIQe4rNFS3Qk5qaQm5bhd rfZfBg/w+uvKqIP65P8WHFyAimJd2mUkiXJx+L882SePv36ZrED9Dj2Gz8BLsdKLZkWfPsO5YAQa K86+cFvI7EhvT5/VgERBvqc9+ARha0Jzxfro3m/dab59hkHz8AuXeb6pHhCTF7nahyETAzLH1L4w NZJUq812b2ECMw6gacrbky5lHZ60BhsWzDJVvO11zrgY/umT3djdWq/c2oP58OJJ084fDcLKCIqa wC+H4sASkIp0nL6zptV6e2yVhIlHRDGd6JHD7FzcaZvhiCaLXb8kxyTusmH/2ehTQ2FTLJ3HVGhM aPwifheDDwLDJmouen0bNs1faAO+rD1AWK6/HZOuVGw9uBfQ1nRPBTJTlEssHTi6cObxLVgIMAZ5 YEZp4PBDSrBnatuAL502qFwa28iTg9wyVHA+01gzpTJHS6gJjpaxUHfaSM+cajjmVDHGT0L4Gj/2 wkzESkIUnaAG3h8jbkTLUP+0P1ykghGnXk6u+WAkmunVL6z/aZdzH0Cx8qaVKUNLzbholxnBMyWE ERmwKaHyGsX0WKiGOnjmopUpSHM0gu9EkkaehCjpExjBo6kp6dG4NBYIOvVyCnVdM4JvUNfgaWly xghS5BI8cbYw5AUTmDrOpgZX8Vg3xRPR9/MUt1ATUtzGQiU8xWdONcRaS43tTvduUEMsIZObuCRE pd4XH2km6YmM/8TQRMt7uokb//juQYNudWRuUPfzoC7kxYTveCWNpRn+0fJi3gb7zPCy/8Mo65nu xTsvejQ2rb/zknR253XkMzOwyvJiQk0ImBYL1dj3al5MaoMDkHGSTWqSEIXYx8YjLDudZeN9lu9u yLqLCPGn2tMLUrdPQtyHmoW3RhXfs1EHu9IvuxBbl4g6S6I7XSTdDQWt5LHfypgVXEasQoBhEDBw mqo7ib9+0/V7XCq1S6DWxL2aeK93Ar4dZeSpnb3dmNQOIKEq3rwVky6q1tebrYEsFR60lB+r72CO ivsQj7svZks91emC97+3G+sh8GB3T2r3ZRU8qjCgTVF4pniF5U8HOpkYPX9fud32KYQdBzZYQeGo pNhG+vzV7tbu+2q9/Rq+HQV22pffHgy9Pc6FdH8QB91BJDtcrLWHWcDtpVmAaibBvbYxQXsaTVY+ q43xCP/0tHLq5RFQtRsCvay4chmMULMrl9hTChe4xTHtEeOdYImm+v0GbvlWwWJaw5Dpcjr/Pm5c 9Haz3z7aP76sD3+Yi1y3Fwu5LgRFO3FdEkzk6ZDfuBjZ5TUD0mOhWg0Jz1y17zpxXRn+Rmh2EqL8 nI/D1csxRXEw4joWpsd7aZwfAEd7p/30O/7Tvey7OmVOZ/PRgiqdzsPdIq3J3PA7G3i9e17+/gVs lL9R/p9j93WfpcEQdzbqUff6aWV8M7vt95XeBVN4U4WuqEO4z5vyrQx19ks9Uvpkv+QbseppQsCx DfYyrIIhoWSplY4Yp7YCQWWl9jy3OxRQiYYkkYFLmS/BONMCvbHl0EGYja2Y54AVGD5XUZyv9yTF X3xeai7jdf6zOnyGTXfj/2YNtnjBZAvmSjF7lZqZSDaeNockDnxaXAx6t2ihS8ZYoUOpDR4OQD8X KwhmjpFkeXaI50Uwnlw9rvdhr4KZQ0uvFQnmON26rE2w6l6BrfL31f5xG7ocTv4El66qmxpR3vJ2 STHCkgrg004NcKcEiFjuCTkITeLZzO032wOcsCFDWFQohNP/EyRYA+limsqw2xfLN0jE1Lx5YEnQ MMBBmHndD1xCg7RSH0+064lI39RBfw53cmFqdOUg81FOpYIsO5X2/jBmmTidSkFXxO3QZ9fF3OS3 z32vx0L97XM/zc7lRG9lJ6anUtEPjNt+fgPTl5kMh8y1CI05bJ2DVjxsKDu5iukcM0RNnY16gqXA KA0JKdnHFpxKw9iOnMKvxRDeyKx3Hr4222g4DMoTA+zdFpunswGlS9WAry8H++fqiwUK1YdrS1Sq Br0pfjHV9dHBMWZzfIKdg8HOwYpNrH0bFg5gsU+nA1hsrAMaVm4AzQZC7awxwCk4dmFTzFvY4ILC y6/j0A4Mch/uZStwefTEGIEz6GCDfb0uJoNMd3zEplESC7TFFR/XJPMLLwriEddvx58+w5BUpf+Q vO9OesTR/psDdPPSaffDUJItRUkujUET3V2o4XoOa+BTmNXwdE/tC9UoCc9MUZJ1hHdTTBSM0+4Y JimhJCqLuhF7INg0jRfUqOAJTyS2XWajM4kWR49GhXez0blLoAro1b+iW3NQsyUz3sCTw+orXsO3 Jo33UV6JgSZTG/usdjZ2OaBejC1VijWk6zUFAv/Vc+2vfoWOzShoxpbH4rOdi2cu9QJ9Cl634SO4 l01wMk/ajWL/TJBMoSgZyASu6l+jd0yN+ZWnhFamDq63+gDA42TYGTpSbNtE6KAiFI4JjP1QrQ8B 3XG0Ryr3yMQmuSenAAMhGTLwTxjyrpgU+3bcJc/OZPkRbnuKHZ39aNt0KzkZKFQ3TKRDMkTcORy+ H9voVaDVuFj99P4hwrdFSDynju4CyeQNJhc0VX5+FoolsFntvu3tE+yPSRnYhaZIsW/ZLS8R76Yv EEMX9QWxt6mrBOgerZhnhMRWpjMNVc0zJCEtR6R7X93+2AqL16R9MU1iWKd16OfBMYBTsLoqHmzI H/yv5G0XDg7jdyMmePCiYqRSCIUBDgfl1KUwukWODj+UsvGl161mmFO2UJNTtiygGdSEe8xYqKZs WRA0wXqDZoaEnrIxjrODbWlIttCDGWULNY2yNcrWKFujbI2yNcrWKFujbD+dsnVLvUGUyygb1JxR NixyykaJHgv1lI2SjLI5hc2csgnc6Yyy4SJz2diDOWWDmkbZGmVrlK1RtkbZGmUrIDSkS9GRVqv9 oxqOkcdg1IefHGutMb/G/I7Mr1/I/JTMlXVQkzM/nvlG+JroGxEK1cyPz30jPPOzcsBz5tcZQufM j5f5RgST736wvsAHwcWkB2lgSPwnAi4Robt2LAz6+NR9mF+QLAHKUzCGCChwjGaE++A3VX6oJE5g ed7g/nH7DYwRx2lWbMvVyFoja42sNbL28chaY1mNZd2bZS2MttJjbDIrNqjJWRZmPGNZmDs9FqpZ FjwzY1mCIsznLEsPJmNZIEQdyzpdicYenLOsXNF2b/3azVnWnTy5Gnlr5K2Rt0beGnlr5K2RtzuT N4mWqsgGmavIhnMXBD+kGXkjNtmz+UI1eYNnMhUZBDqdq8hOV5aJvIEQi8lb6MFV8vbDLkcbeWvk rZG3Rt4aefso5K1dkzYO+EE4IF58TWrOrknPFXjc9Pk1qcN6LNRfk/pn5m6o3M4uRS+6oYIQizlg 6EHjgI0DNg7YOGDjgI0DNgVeI2/vhbwtjLTEhF+vmUOqrznzbpAq924YmB4L9d4NA8tuXxk1aE7e rGEm826QZWnIQg8yh1Rf07wbikVs3g2NsjXK1ijb70vZmtquMb8PwvwWR48bcB49ztfkzI9Jkmc5 U1qPhfosZ/6Zudqul/rN6HEgRI3a7kjvjj04V9vF7oZYdskVIv5JX7JlvAe1XdOvNbLWyFoja42s NZbVWNa9WdbSgG+S0zx6iK85i9ErcpaFZGJZvlAfo1fqPOBbp7LoIUyjbs6yQIjiPomQMBabrhOx xoacYUClenHqd1icUnWnv/GFHtui0VsQyTwIFCbn/ul5td58VY/rkFN2DZNj9/IM8zSEHy9OLBg7 ENq81B6COav+UZEBoWnYGmlrpK2Rtt+btDUNW+N+H4T7LY4cR/uc+/manPtRYTPuR5XQY6Ga+8Ez 87tVpF0W7Ffq3s65HwhRo2GbGcaFHjTDuKa4axywccDGARsHbIq7Rt7eC3lbGvxNktyrAWrODOO4 zA3jeqrHQr1hnH9mfj3KiB3euh4FIRaTt9CDRt4aeWvkrZG3Rt4aeWsKvMYBfzEOKJZ6tprcOQJq zqKbiNw5gqjkHOEL9dFNVOYc0XXO8Lc4IAixmAOGHjQO2Dhg44CNAzYO2Dhg44CNA/5iHHBheGLP +PAZBzx3k0Ca5gZ8xuixUG/A55/JwhNLm0W4M46IzIBPF9mgxR7MHWSh5jUH2R8Wnvj9w3oz32vM rzG/xvx+b+bXKFujbPelbAQtDkps5j4XYbYOvMv0bCy/uiVdurr1hXq1XZdd3QrScTenbIMlMlPb saKr29innLLR90LZhsSH1PNarw7rJz8lpv4W9e4R7xzfGwVsFLBRwEYBf28K2JR/jUl+ECa5NLSx JHl6C6g5i5GCUR4jhSg9FupjpBCVp7ewLssA21PL50wShCh60yzaycgSCetOXTne+8bu/rgYKY1W NVrVaFWjVb81rWp8qPGhe/OhpdGCJRnO+NBw7tHKTO7R6neUsVDv0eqfmfMh3Rk750PE5dGCQYii N130aGWmGcSVG8Q13tZ4W+Ntjbc13tZ4W+Ntd+RtdHEkktyIDWrOHBmQyG9ECdFjof5G1D8zvxHF XZfxNiXdkN2IoqLIbLEH8xtRqGlZHhpla5StUbZG2Rpla5StUbafTtkWpmfwZOYscDA9DxzMMcuz qhKrx0J9VlX/zNz3FGGZBQ6+kFUVF21UV7KqQg+uqtp+mBFbU7U13tZ4W+Ntjbd9AN7WLM8a/fsg 9G953og89AjU5PQP6z6jf9hiPRaq6R88k7mdWp7nZeVaz+kfCFFD/2JX5KQH5/Qv/o0T8wQS/F55 I+5B/26cNKIFHWnsr7G/xv4a+2u0rdG2e9O2xSkf+FnKB36e8sF/mJy24aS184V62oZzrV2nRPeW 1g6EWKy1Cz24qrVrBnKNvDXy1shbI2+NvDXy1sjbDyZvS1M+CJR7e0LNjLxxJiXjKKvhTo+FOvKW npl7Nzhm9Zy89QOahnpLQiwjb2MP2pVrI2+NvDXy1shbI2+NvDXy9l7I29JcDQLlrqlQc0be5hem UBMuTGOhnrzlF6adYuR1zVsSouhN10N1jF05Ktxid1uojsaHGh9qfKjxocaHGh/6JfjQ0rwFAuX5 S6Em50PDPPk81ITk87FQzYeGPPl8hwxSb/GhoSz5fOzB3OUTaprLZ6NsjbI1ytYoW6NsjbI1yvaz KRtemrdAIHdG2dwZZVN6yCibslyPhWrKBs9MayTiUDgStH7g0g6z6GpJiMX3j6EHV+8fm/FYu39s 5K2Rt0beGnk7ozbN77NxwA/BARdmHOgpyyK1hZqcA1qTc0DrEgf0hWoOCM/M/T6RQNk1puSzBKRJ iMUcMPSgccDGARsHbBywccDGAZsCr5G390LeFqZH8FSNnpE3ek7euMzJW0/1WKgnb32WeLSTwtq3 7lxBiBryFrsiJz04J2/xb45BO0wXaz5U0A5zPWgH/tuSVKbtyrUxtsbYGmNrjK0xtsbY7sXYFiZG 8PSEnzE2fsbYHDcZY3N9r8dCNWODZzJ125nXgHCsmzM2EGKxui30oKnbmrqtkbe/RN56NsghTQw/ /bc6QOx+tdka+xSplxMg1lC8jb0rtnSr0wpig28utOXC+kmb8wDfbKhoxaUoin7fCA35NmRgo8UM 613t8LfcnpnueART2KDH0bGwXl3xCeID7fELI+n7Hb0/2+OzmFw96vr5Hg81YY+Phbo9Pj0z3+MJ snS+xw/9MNXKJCGK+zSzhA81zRK+qWXazt7UMh+AaLxziGhqmaaWaWqZv0zZFke/7/NIXFCTUzZu +oyycYf1WKimbPBMFkbVnz9evUhLQixTy4w9aGqZppZp5K2Rt0beGnlr5K2Rt/dC3hbHwO/zSFxQ k5M3Oo88ATUh8kQsVJM3ehZ5gjKF3yJvtCzyxGXyFnrQyFtTujXe1nhb422NtzXe1njbO+BtS8Pf yz6PGAY1OW8jTGa8jXRUj4Vq3gbPZNbrtn89gmoSooa3HS9D48PojLd1+kKh4k3vhLc1gtUIViNY jWA1gtUIViNY9yJYS0PUyz6P7wU1ZwRLqJxgKabHQj3B8s/MCVbvev0mwfJCLFaMhR5cVYy1/EKN tzXe1nhb422NtzXe1njbj+NtS0PpyzwmV6jJeRuTJONtTGk9Fqp5GzwzzwtpBiHmvI0T7ea8DYRY rBgLPTjnbbG7Py61UFOMNYLVCFYjWI1gNYLVCNYHIlhkaeB7mcfNCjVn5v6Y5eb+xOqxUG/u75+Z e2hS5jKCZcgsDGoSYrFiLPTgqmKsWYw13tZ4W+Ntjbc13tZ4W+NtP463LQxW71kaP+Nt/NzSH+nc 0n+0rvKFekt//8z8QpPL7k2LMRBiMW8LPWi8rfG2xtsab2u8rfG2xtsab3sHvG1hnHrP0voz3nYe EY0Zml9oWqPHQv2Fpn9mzts8oyFv8TYQYjFvCz1ovK1czhZeo5G3Rt4aeWvkrZG3Rt7uS94Whqz3 VO0sNho7j43W41zp1tOkdPOF+nC2NFe6UaP5W+QNhFhM3kIPGnlrSrfG2xpva7yt8bYiVtMShDf6 9yHo38JsBp7snUVXY+fR1fx3yegf4oMeC9X0D57JdHeyM2/RPxBiMf0LPWj0r+nuGgdsHLBxwMYB GwdsHPAX44CL0yOws0ht7DxSmydQ+f0tUXos1N/fEpVHakNv5BlPQiy/v8XnkdpaIJHGARsHbByw ccDGARt5a+Ttp5G3xekR2FkUOObylONcz43voCYY38VCbcrx8EyWjlSZLD2CZnbq/pqEKO7TLB1p qHktHekPI2/vH4/b1W2jbI2yNcrWKFujbI2y3ZOyLc6M0J0FgOvwGWVzc5M7qAkmd7FQTdlcbnIn 8KDyDPKYmDllc4tN7sYetDvXxtsab2u8rfG2xtsab2u87R3wtsUJF7qzuHIdPVe1zQP3Qk0I3BsL 9aq2PHBvR6V8/Z40CbGYt+nLgXsbb2v3pI28NfLWyFsjb9eoTbOVaxzwQ3DAxckburMYdR0/191x k+vu+l6PhXrdnX9mzgEJOlK0qxwQhFiuu+OmccCmu2v0r9G/Rv8a/Wu87VfmbQRrG4F4Yz20JIg5 7lzFvfo4/I8uzi3RncW66/oz/jewnP8NXeJ/vlDN/+CZLOkqU/It/gdCLOZ/oQeN/zUdYCOBjQQ2 EthIYCOBvzgJ/EDkbXGCie4s1l0nz5V30ubKu0HosVCvvIM8q3PDO9xn5E3RLje880IUvWmWQ3X0 gyCsO3XlyNlid39c5lWPBJ0oQIK+VtXWmFFjRo0ZNWbUmFFjRo0ZTZnR4hQO3VkYuC4LA8e5JEig rIYQPRYqmVF8Zq7WUo68odaKQhT3KfMi9TWveZH+MLXW+8fjdqPZKFujbI2y/d6UrRm0Neb3QZjf 4vwP3Vnwt86cMT8qbMb8qBJ6LFQzP3gmM2hzlr3F/ECIhReaqQdXLzRb8Ld2odk4YOOAjQM2DtjI WyNvP5i8Lc7e0J0Ff+vy4G9csswj1ddEj9RQqCZvLPdIFUziYU7eHEP9nLyx5R6pqQdXyVuzRmvk rZG3Rt4aeWvkrZG3Rt5+MHlbmHahpygLAxdqzu5cmczvXDuqx0L9nat/Zq55Y0a/nnorCbGYvIUe NPLWyFsjb428NfLWyFsjb428vRfytjDtgqdq9Iy85bHguOScZ+SNd06PhWryBs9k16aKv0neQIji Ps0M5kJNM5grFrEZzDXK1ihbo2yNsjXK1ijbvSjbwrQLnszwM8qWh27jss9Ct/maGLotFKopW38W us0ztP4tytYvD92WetD0bY23Nd7WeFvjbY23Nd7WeNs74G0L0y54ltaf8bY85BqXHc59UzuafFN9 oZq3wTPZPal4m7eBEIt5W+hB422NtzXe1nhb422NtzXe1njbO+BtC1MleJYmz3hbHm2NS276/IrU YT0W6q9I/TMz3tbrweTpsiAG2vyK1BSBymXeFnpwlbc1z9Jm39bIWyNvjbw18tbIWyNvP5a8sYV5 DjxVG87I23lAOP8vGXlDMnmW+kI1eYNn5uTNWaXm5E0I7ObkDYSoIW+pc+LUg7fJGzo+9UHIW2NZ jWU1ltVYVmNZjWU1lnVvlrUwIYGnFOaMZZ0HXwv8Y16DlR4L9SwLq4xlWaTonGVxSVHGssr6dFlF NqVS7WqzkbdG3hp5a+StkbdG3hp5++nkbWHOBE/V3Bl5Ow++RrXLI+daqcdCfeRc/0yWM0HSN11A QYjF5C30oJG3Rt4aeWvkrZG3Rt4aeWvk7b2Qt6VpDwacBV8LNWeRc3GueWMkad58oT5yLsk0b53o cW6cdkbeQIjF5C304O37zUbeGnlr5O2V7vVskEOaGH76b3WA2P1qszX2KVIvJ0CsoXgbezdsCfWd EbFzj7sXD9OeC66UMSv/v4AEQr9o8UQQveNxqYdNCNrZ2G8rtdluVrsnBVyJw2Lvylu8ISd5V9v+ b7pnLwyY6pcWt36HnuzZShnZ5eoVpPMa36+xUK9w8c9M9+zeCOZmjoC97gy3mcLFC1HcpxCR1X9A PUwibB07d14TDyJOPlSMXv2eHQWKC3n7+JhINodk3V3FIcfLGSe4DVP74pZgfpndFMmBO9izntfP dmyCweG+Kz7b334n9VJ5UB+lSscLIkIbtmRRx0Y4tiaC3v77Ro/tCNhKKzLqcb+WRsibtoMqm+kd 7HsAv6mBHg7gplRZ4VsQlibATC1Q+FC8dDb+ELz0ux9ZnB2kz884UDPTHzOJOGYoqyFWj4U6vEzP zM84SPiTx2tnnCRE3RnHb+NC01MPLpxxzICZEHRy2Ek1H+OME3uYlAJGHeyolcNEBM1AcWjCdqpp p5orM8zIxD9trpLmYY7VNEV5mmNnjREKQ1Tx8d7LyYiwjqM4I9d+cm/sn4dR045N1bHoNplkkXKd iV8Mnl/53W9/2L3ocGQIe05FalvlYktfnuxTOvWNQsFg83K9/zs6UyHiUC9GgmH/DKu+QzCbu2Jl 6kc5TflRkws9PJga5raH6Vk5386zsCpQE8KqxEI1OzgPqzJww2fsoBMDl3N2UBhWJfZJ6IcRjsKR SYNW13FPExxioUZBQgoHPyAFQ+9BfqIKLUGpBaepG+Jj237b9ntPFd472n8F9Tg6gvm356Ajg01B A4JyVnzYE7RHV1WIoqkQf8udc2HsfyayQLKx6UHPtzqkabZzImP0WKjeOeGZqWbSCwGFIANDcT8T WtIonxUnIQredNIx+naJcOLUbuzk1BPyeuE+5+q227Xdru12NbvdDbnmDU/AbQ8uGW6jVbJtso9W 7e3RsE+ElqT8aBvxp+enh9PvwoUgRQvd59iA/RIbjVlEF7a+fjiLA8rzDIpdnzIo+kJ9ZNB+mkGR DAZ3vRmPz1iFGtorfqoZhSjfiPGpK3ByJbqfFKKeO9ks9Zayh8EQf0Zm8WqUWVUycRccYRn2cyou 4N1h/eTnw/6gdge/K2/8lN+A4SIOuFl8peRbxPZCi/bP57Xf2KFB0JkXwzAjTpIwY8Em9sVjTGx3 +zJKqVOAUAkBQodiRGUY93QmKOzLwfAz9LhUfwcCcvOKgMAa4CKsfJUPFFT7cEW7Ojaamhz3QlZ8 UagwYTLd8Xn8CXpFpuEb8CLFyCjToNL2E6+OAxuCgVLF6kmFNHf57TOquny+QROEDP0l++jj11eA qK74cxFs8XDd3pqrGntrRHqGWdoEv734sd4fVoHiqUQWMUhnK3pL+3hPBJvOavfvVVotwaq8WB/8 QayvEXvVMGGoMEx4h7YSfqpxO5LA0cL8eBdJAf54MWrd64PiQdFMxGAS8MUCU6WAO6z4UhJxl6bH Xj953rs/7LbfV3pn4Wo/UNViW2rUIdznTaU5UR5VGvUoWF3858PLZv0nXNRY9eRxdWOe9p/C6gyH uY/cww5xcmxlZ/VXIJlh2pd6HPg2rDJnowRtxVFiKEwDUoyKCstw+tF+Tq6A/R53MoyDN4QsbYqK ARk2XtrBbE9irTbbvdUgXDhHl65JKgxOABtaU+vtaHqiQTJTepaiQmMbBYsd3JjDFr4etKLLxVGe mp7EGSfUGoxajyfFYtcRxUTfJfzKGgpzs/gGT2E3jKNuxivJcCaj5bwIO5fMjI4t6JpvhRRBhqcW vqzDoSfYB7FieGO8E7EfX4CQrT5Zf3QGUquDrVPxwVnhgQ/pJHecyx0QO1F8rrzJUe6zejSvH+W6 hc60THI59NOjHGecDrn3RWeGvMZxPRbqj3L+mWs6VUTjJSQh3Ia/EZqdhKjSqTIhTTzKifEoJ86O cn4AnB8Ay6w2ehjNo1g7yrWj3NKjHDdiNNfch4MYNMJhtKipuM/5ZQ5zrYnWxK/dBJVoOBKnx+32 S4A1Cxu8LcYhihiNtMWf7cZ74OgJXqoCopJYG48+evv83TOw7dPqZW9BxwEoVsxzJcU2Okl/tbu1 A03/10C8KWg2+lIA8kS+t0ciHwy+x+OFC0a5olQn7luSyb1peiAYEbHcIEHIgadTz+N68yX6R319 UrBTCuDLqtwwv8MpE5IDivm4/vQZPlkPh/G+/DAwDDRSAvf8cmoFhqeUufs2jFDHY8lpbOKFSvGe czvi3AubiPO4/4UruR97A1JAmxeb+BM6cYnqUg0Tc57LOM7dmLtBj4V6N2b/zAXaDIUumSLoXgg9 NSuIQpTTZndWmHlC9Z3s+wEM/RixiD5oZi239PjKks+yhDbDy5K+A4Ju6Gjh/wy3zgKWW5kbQ2gL m66L1877p2d/VP6qHtfByn8NE2338hzWcLjRLDb0f/emEs0/6kfekjdnqwtI/MfuZbOxu2tIDAEl gh6jGonxRIGRTvRnd9E9zp1Te5qcU32hPrs4nTqnCt0rJNUoBA41EjE5nGpGIeruogfcDyi/go5I DP0W1/+J38k5lWGUAiCtVgkC0nEe1gmtuUa7F2oy3HMxUzSkayutNtrCxA2+H7xY206c0EOJRoTS Go3I/XYNySP4gGkWYP0qmCKBgIA/tEp9MPqvTXQQ0tUoIZq+5kfqa9odcLsDbnfAt78DhitJ0U+v JE9tBZ0IK72v8y0pM7mMPLUjYftQ5eY+N9E6+VZon67bjm0El8biTwZtqPHK7rPaf155Bn1qLMBD V3r/8A5VWA7r2ZVvmkEieIPL4lHCpk8L2T2++EE6PA5HS2NXozJCWGuWbIMP1i9bC+ZHEMJw5R7V J1DYEFR1hXwzA2hL01V0oB3gC7I2QMCiLW2nAfpU+Vn9loeyu+VDv5kRMhXamol1gZfMHsazFodZ iyuoxPvRwf3heeh288d6c/HkJ+RDOgBWn/x4dnUNt7t65g6klEJzR1pfQ4IjbShUugPFZ65eXbPo v0O0o6eaUYiaq+v0cJcXkqcQduEym3fKL9w0EiGYSLxDuM/J7wbXEwizfjjp3tTzWq/iCW2qe+sC PlfwoyFp4a+0R1RYhe2099EFbAfJdvG/xJa8HcbaYezOhzFGrBLixHzddneaY9HGtNSBzDc1YHrW 1POjHZ0LQsyoYgvEjhmS9lw/oz7vtpv1f9vVTr9UG1ZSP+gp4Pl6A3fV22e78ZPi61rDnO8BfEXp tQ1FBuF4iLFffRuhMegdDT6pXbm6l2g+TtHT3aD/fyowVdADYVquublR7C/EkGN4poX2H/Kb2oUI 4yhGhy+PMY6ovA47f0uOvEXpeOPY646YydiPkc+DAh/3xasHaabVeTuoqpXjNdd//oWLrvd33Nkf Lh13GF563Bnw8Jalrj9pSLgLmtdYrMdC9XEHnrl63NHxlENUN0wsdZMQNcedcRW/bqnrBwAsdU1P rFEOH09Y9zI5uCWsNLvfZvf7u9H/W+9aiPRcx33CrTdr0KCCt+T+2/qgPx/1usV++gTrdCzYWL++ Uz4fkAq+GC4OQkAcSn5U4xwKCt1wrihW/XkM1eQY5/6g/Hzcr+M5WT9NtuYiD4jQoNA6Bc4PTfkd afyApaBFFMcmNuElel6v/nxU41TEpJjO3i1jBDFoNuyTpRvaUsWDj/nQxfPJl+3wf6w+rJ4jAgDK F/s+EYEsivMTyPGRF9clFZKeN7rTd9tBGJG9PwWEw2p06DI1AetuuQJveMq5dYTy2x6aCNJjoJvh 8YvfU61+8WeBMGTBvFcURwJCWqfgXqGljUn2YP8ejeVKCSpBvdRRJvv4NVz6pMlxZKjF4IBUP8Sm Ps2bUSow3dKvSJAUw6hbAXc/M2msg8NlX0pooKl0Es8bivFuyr+dlCPgrGDMYTGeGhOwpsVPVvh4 Ga0wx1kxlzDm0yolDH4daT0B+jRXAXdAN49RRUNGirOGQDC4xIUJ1hfPet9WYjEX2upZVVvC+HPP qS2Y+E/WrF+eogFa4OMs3AWWLyX4xTtmtT9wvyz17nu4LOBxeaNiZi/9UTqOGoT0ThtHjKtVLI4c kEojv4shws12uxtnVXk7xgwpYYs2cG2eQhmFhoK7hSwe9Pe0JQrdD5MJEFtJ8hBXpc+8k/62YyK5 uez0SzqIfQq2USF6FBblCel+PReMpA9R+qI+JGRZWKQP4YMjE33IRJswUWDYuecy1ATP5Vio1ofY y57LgAMYD+TkuTwxx41ClPQpKD06Pb/s9cxGvKEY8SPh+0QU4o7cKytN6N/xGu/Zz/B1RF4Ac1ZM +z6GImTQ8SC/UZvt/tHa59FGpyiHxShW2OFHoaZNdQCiRbbgN9Wg+IYGohIozLoWDqvFaHcTZDCf 9bM2rzhnSSaWhqebGYZMVsgssCuTJIsTy5TWY6E6Tiw8c1VTinFwzpKDb3CiKY1C1GhKj115HRAI B02ptgz1iiQ86fF9ckx+kCXddJtNt3m7JghmiiQr1ecXSEQZI7SqTXBhCwF8ikeXcK37Ua9pVsPL fg0qn0+7bTTG7aG5vijtbWxPMBmH+XGrfHvqERxf4OwbXL9cMYm8XeTgpgsu1wXfPPqV7AY7qtq+ +o3fHyvASns85hTHpb2lztV3UqiI8cGJzJPKHbiVxL852nGUQsttAnO9Z33pbfIY3Tj02O1yGd0s T9MNYpjdsFctpNq7VEt8Mq/Gxsaok0sPH0riMzMNJVSmhBiYyWu6Xo+FarUEPHMpMkSQRjE4fBhC qZyrJUCIsj69Je/pn7hWPea9e3A97VUvRyGKmNWSw8dtJuYd89r/lsbkzXX4tzwWvY8mqMRCodGY 4tGvitX+ywCbNywyV2zuKixj+sSVUktwhfsc25MmaCGKD0HNMPvHG2bfCSmb0/mSUbuRnymIl3y+ cvGOZ/ZiEP8APqYQCMrGHeIYdmm6NUMoKFp8aCNW9Pb4EeJeg0W4Hx7KdVZWMn48b0MrbFSfqnK3 F0P6CPX68/oxEQ4g84MKXzKctSp24796tvCd6uadel4HLVCwvSveim9yRHncflpvXjui9F2/NHid xx06uR+5fHMqBMlrpNZjofqIAs9MjyhyPKJ4aawT4YhC/S+eXhw5CVHSp55ibbrQiuowg3yzA7Xp lgW6G+5ow1Kf/o1CBtyDXeeMUboEHRccUW4aWcwv3sDQJ0u3w9VLN5Dg6cINsQx+1sK9yXrxO8nh 8P3V9bLwSN8TSa1CZ5YGeD7Bjck9L4xLnhe+UL1e4JmLlgaMCq3FZAWdllIUorhPQd6YzCaaKUDO aYRD5+iVF4S/GWvus15unF3nbnuy6hJ7i5eTMajHt+3uy6gnltWK9b+sVkeU9dwkTvmYnOHB4DQd MIptyXz3rIqUI3bMN6e+WzN2kIJYtNhOtCM6xGjcrCBt4BiJMOTp7MrvhQke7DBnuMEqGkQpj1Fy U9ZNuMMdjNOkXyFFRLmpXUdkOmbORgdVjQ3IgU6JGSdtFAvy64SbLNgL5FL1ru4L9gIputwLzx8Y x0K9F55/5gJ3muwFk1DAfCpEcZ/yvWDQbrYXXHpB+Juxpu0FbS9oe0HbCz7eXtCjxXtByblAMp7v BdzpsVC/F/hn6s8FIERxn9q5oO0FbS9oe8HvuBcsT4luJnsBubIXBMY8r7FSj4V6sw//zMT0AilG pBuNPDx4h5ouRSUMNaMQxX3K94Kg5XpjLwh/0/aCthe0vaDtBR94L1iaU3XA7tK5IPdDxHlgWktT YFpfqPdMpOTy/RpIY5n/J9JDSLIkDTkJUfQmrgZm0UPv4m/C+cMLBLwgQhT88ZBsDOGfOv0wAuxd 9oLYrfE+W283q2+7dXCAx5oEcxhdlM311hbubU9pe0rbU36rPWVQ+8+vmpUL6veUaLpRq2siDm6T j3tKqJnRf0BkN09zBTUhzVUsVO8pbpbm6rinJPofEg66TvkXDGC8gclJiII3HXfFYI9hqEh9MiKc OCBHTKqxOoQ292cRBjfXg8fV01N3Ol/cNkVNkD3aVV9PnQe2SuVOg82qpNCY6+ebcoWN7A/xT3QV FnjKQ1qd/W6ggxOjKRfrYHkghrvM1R1nQUF9TQwKGgrVru44Cwoqp67ucUEbxyyZeptEIUr65B/u nZpE0Ij/6Ujqbi9O/QarPw2genpTGIAiUrsAFsLLwgz9NsBUCEwzzU7/GWFqVZgQaxmC8A5mfWpo dficNrCQx8fvrxWysRSy7HKDqK49ammY+l+ODXQBVYqTPfgGusjkvsxkKG8BEU3BzDdWpp6sPtvH 55huEDpU8qVfjP1qXt2XOdhSYkLqo9CwYejQ1N0LrBcRyw5nMncAswPTY6H+rDew67Em0PQyKMiX LA9AiOI+kZhOBPyBUo0Rs1WmkYinP/hjcTzipZoi78YlSUjef7aLG1pqKzSkVMXg9xmMmsEDfxcc 8APalDuiokHIo79C8CMdd/rgl9GSYb7d8d/BF+22Ya/fi2cbIUOvErO0vj/Pfvo/PR9O0VoULAJX /O2Ip0DDhfZSsgAOE6E4D+5HucxoKUBrRPoA3li/fNaZIcVNCM6N5tNu5ttIWRBKVHgkvjPfxl82 VaZvyDPc4wC5MZno6nH96fNROf2Dg8MgaRSKfnPPLweIouObICEeTHF0T+nPVOQoyM5CKyqGOA7x lEy5+kfIFHjZA8F682WlNtvN6uuTgtgHAma2Ks7EIzRBajY8R35ZEz/hpvGdKCcjD1l/et5t9VNM IRrDOhTfJijcmaNPYOJEIQJ0keHd7VRA/uHH7ac/Np+unUDlcov0AUfNMEEp2qGvAauSeXhCQ/No h9bosVAf7TCqaVMNo8pxRUw6IFJ5oWYUorhPQYnlV4wV+tSDePsoR0U4Iad/yvRFd4uDCgKNt41u /fg8Jq6iIdQvtcV4Iig1/RUNLmW6aXDfjwb37eULRsRxFdc6FzoX4wUdly+EL81NAzTHeU036LFQ rUCCZy7FCwJptGFzi4VjWicQouZixwjZM67cxWClJAUrhVdCsFIwUWadTvYJuMhcuQUrbcFK76YJ eS+6i3fRBJZG6JSw4rBz+5CFYfDg/tkT83ECFUe4xAPXQ1yF640/rxxgyUSTnaDqwMUB9u50NPYC DikaoV8nfsmN8smYU6OUPGJpO4cmw6YfrdqdDVxxJD88dFzHQ3ZsKcgF7cGFEUxwV4pbt4yh4meH TBFsL84OpyqnBxOOn/Te9s+D37qP36DOBMgPGaYpqIc9jG3FEWMsIEtxNhjflkvJPtIci83BgMka dTzBPelGiyJ/unlerTd6tV9vPoWIsdTUqBMYJaYjqTF12D6t9WqzPazd2kM8UJ2V/uwfPGq8i63D fLt9yqz+Wquwu5WegElnCI8LIjT2fRXTrMJRmtTskre0+rtlW57eEC7SBVDIvgIN7rd+Q9vbpwBE MRNLRX5bSpKqdPsNxjuIVGwoci/lISMExz0lBql+F0L9LoN/y1vGO2qX/7rR5CAwSdag+wgYIIgK tw262I7gF4teuz/AkecVcwYpgplhMIypUyYJ1E9CQ3VjUokuOz5myiRfE5VJoVB/Gp0pk6RkWg+O H/P7xppO9qeaUYjSPlkHtoRSwc1CeFgxiEzLmet7hGZ2SuPfwEjwYGHkD+PiPqbrsemz6Y1U3fS+ XdTWX1GLdJMlZ8Mx4NUlRx/SyjsuuZIOO+q6XAEkuTxTAIUJO6/pqB4L1UsOnrkYMDqllwnLXjs6 VwCBEDUKoGOK8tcUQDAAoABiGhvi0HHdlxD8pgBqCqCmAPqQJjS3NTTyhxGT9CLPL0rrA5xOd2Em ctjAijL9pZaETlYD4WQ62vRgEi5EizhH4r+mN4mjxy4e4hVmoOdd+Z5MhkRerwz8IH4t2yWBSg4j rtp2yV2xXeq6Stulm2VFgqboaCR01ljMxfKzcn3/yhmWEO1GMHsOZym/Y29hLuioff55Vg+dlnYq 1/FQIAOFLjfjfC8nXj9EfZ/OA1FrGKk34HsJyAR98h92Y9bbP54uE2+x0HdG6RAXONFNcKnzbUmU mUl0mem+r4mm+6FQbTjRzUz3cScHho+W9YxDjbI91aeaUYiiPjlu1OlAiz3fRp2IVBwKwUMv9fJ4 tI8jgaKTHeNF57QlpvvQdJiSYX1EAjrE9NQEtkT2sw1cKetSSrLV6mlroohwsWJrDIY8RNGgTb5A ZIPpUY1TtUCJtu222u73k6ZQhYEfaOIE4B04UVtIMb6NurjIiMuvUqzoIsuaN4SqmvmVPIv2amOG 7Z9X9QIYd/whqQeqXfvkxK7rGEVCZIZYmWGIr4mGIaFQb9eVGYbMPH5ZEkIK/jD0Anu4YUchqvQC wXfXodhL66LHLx2UOkV/t0Yj17suKAgQGWO7o/5OnkXh9WFO2U3O6nC4BCLFu2/zzf0wll12ePHd +Xvy27+4gLuHtI6rw8ZnhplQc+ayb3mu2AvbfyrUuwb2FxV7xwUMK84p5mtkrz01E0chKhfwoFx0 BIzpGyYLWMYFPCgEqn0NC7gPDpDjU7/ZAm7q+V8FCqR8SIhQCQW0N5af7+VkvvnSLMmir4lJFkOh ei+ns8SMvXGDdmpy3oGaztPV2QkoCFEGBb4vRo2hBd2Z/TU+FTTmxp9y5jba4fH7QEHPBjlePces bZBoar+CJLfJj8AFlfJQnMjxfhf5A00M3z6rnY26qbC2o+KymNLKgQdTs+f1sx2bYIB7XTHsUdsl u6tXwpSgGgMIL5UOGeaCVImqExHasMU3LH1nRPyej7uXlI8XsNT/r/F2gBarNETveNKx+EkR2tnY b9F/Zvek4DzDg/F9eYu3yWqGJMchWFW4t9jocbhEuEspxsJb5zO7sVuPF2Ecqv36kxpnFQ7ehuWW kDBYHT22NBkuVDVav0zsqTd3L4IikZW1LgpUEPfm7qVUUDPNa7DSY6E+X5h/ZrJ7Wc/vumMALNi9 LMLaHuMZht0rClGze3luKjx70m33artX273a7tV2r3e7e/UPaROr2704t3SmRw01Z2oYg1mevY9Y PRbqs/f5Z6a7F6bYWjfdvXxXrFLz3QuEqNajMsJSny6qYWwnudZIwj5OER7E8ammhmlqmHeohoHN +4kd7B/k7x2/DAXiISFCJRQwQ+caWag5D6LKTR5Ete/1WKgPotr3b1+pGIUehr6TegwBDkJUQ4EK FpbQp4tXKq4T2DjkAAoY88vr+NRvBgVtAd9zAU9jIF/ey1lawKL2JOqkmS3g9Gy2gAXLneUFT87y vlB/EuUX70RdZ+zAlZ4u4J6zeCcahahZwHlhtAkh9gEJf4YIi7xzgwnvhjtRPe7l5l7O8m3RfZhF 98TeXHTh8gLWXu2i66aXF+SK+sfwPM2d6VKaO1+oJ9Cdu5zylBvCUPJQst00JmkSokr90xmtmThf fU3909Q/N1L/3EYv0hQGtSSDo4cEe9UKA1mg7oaOZDUkqbt9od4HkqjLaT2neGekneMdCFGn7va0 pKm7G97dEe9+XX1y0//+TDjHDwnVa+mrMnP9rz9EKUXyhBhZlmZfE7M0h0J9ioxZluahI8Z1fKLt hRrM5vrfKETpmZG8cmbE4SyK3HhmDN2FwHKdHpU++l4R+m8a1bDpbD/G6XP/dTNZvv/zYfey2axT tMS0fMlDWsVVy/fEdvxJjUAGQtejbrCJEWFIRThbBaEGXBV6LYdjTfmb4hLCUIhvGsbcFhyCYKAe Znj6Yx4zbSDTn3I8Fr4piMmE9KfPjuXrmF8IZvqDwpvioWOkP3npnMJlgfNY2OVEqa9e/Ab/OouW Fa/Aw1SNgS5K21MCaTnjX4EpAkkJvt1EFWfW6PnoQRLjqIXLdLPeHb6v3Ha3WkN46BAcTFZE4ZIp /rlfQaG9EfYCUpVHPxODYNNgapABMfDiMabRkz0oow4KXJNCLOPisHZKGj6N4ufWHqDVensMeMVC HOreFbsoSd2liGjP6vB59RxzRwY32GKXa0+GUhsjGRrFqSN7twxpd8u2fAdRcuvMO1hL97ouOkcD 4TsOUshLUB6DnPf8FIN8bGQh6fvPv7hvWPMW7aOL9g3ncRqfn+I9vGXHbknyU7zSeizUn+L9M1e0 lqcjNWgto3xYHIUo6RP8xr1QpysIX7CTF8R2yfRvQk0fvDt7JuWdrv1D02FCgKtxhD9IRwABHB/X e0AFYgEWaLGTXjvG/1SrNdSN8f3ikI/neGinOPUCEmRI8SFWgBDjlkhMHzbF4nZupQ244eGkneHn LX32r34DzNlCMEdOnl9BnYG5RLkNl8TJhssXqsEcnikEc5AvgTkIUQvmBpGxwGZgDu2S6d+Emgbm DcwbmN8WzAVn6ct9fVIryKgDOAwTSBTHNfi9NoT/aL/2a7/2a7/2a7/2a7/2a7/2a7/2a7/2a7/2 a7/2a7/f8Pf/A7zHA8gAAAUA --=-9DLhnUr6//hYyzbOOvWY-- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Zhong, Xin
2011-Feb-21 01:51 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there's no debug info... -----Original Message----- From: Maria Wikström [mailto:maria@ponstudios.se] Sent: Friday, February 18, 2011 7:32 PM To: Zhong, Xin Cc: Johannes Hirte; linux-btrfs@vger.kernel.org Subject: RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page Seems like my reply got eaten by the lists spam filter, so I resend with attachment compressed. Should have thought of that :p fre 2011-02-11 klockan 12:39 +0800 skrev Zhong, Xin:> Hi, > > Could you paste the output of sysrq+t here? Thanks!Yes, it's in the attachment. I tried latest btrfs from git (last commit Mon Feb 14 00:45:29 2011 +0000) but it hang so bad that I couldn't get the output from sysrq+t to hit the disk. So the output is from vanilla 2.6.37 // Maria> -----Original Message----- > From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] > Sent: Wednesday, February 02, 2011 7:35 AM > To: Zhong, Xin > Cc: Maria Wikström; linux-btrfs@vger.kernel.org > Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the > mmaped buffer of the same page > > On Friday 28 January 2011 04:53:24 Zhong, Xin wrote: > > Could you describe the steps to recreate it? > > It will be a great help for me to look further. Thanks! > > It's a little strange. I have to systems with btrfs, both > Gentoo-based. One is affected by this bug the other is not. On the > affected system it is enough to do a 'emerge dev-libs/libgcrypt' that > should normaly compile and install libgcrypt. The emerge command is > part of portage, the package management of Gentoo. > The strace output looks similar to the one from Maria: >NrybXǧv^){.n+{n߲)w*jgݢj/zޖ2ޙ&)ߡaGhj:+vw٥
Johannes Hirte
2011-Feb-22 22:27 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Friday 11 February 2011 05:39:39 Zhong, Xin wrote:> Hi, > > Could you paste the output of sysrq+t here? Thanks!Sorry for the long delay. I''ve attached the dmesg output of sysrq-t from 2.6.38-rc3 and rc4 as the behavior differs between both versions. With rc3 it don''t need any interaction. I start ''emerge sys-devel/libgcrypt'', the tasks hangs and top shows a svn process eating 100% CPU. With rc4 the system hangs too but the CPU load stays low. When I try to cancel the emerge process (ctrl c) the CPU load goes up and top shows the svn process together with a flush- btrfs task, eating the CPU.
Zhong, Xin
2011-Feb-23 07:27 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
In the dmesg of rc4, I can see svn hang in shrink_dellalloc and there's two flush-btrfs threads hang there too. Josef, it seems you are the expert in this area. Could you take a quick look? Thanks! -----Original Message----- From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] Sent: Wednesday, February 23, 2011 6:27 AM To: Zhong, Xin Cc: Maria Wikström; linux-btrfs@vger.kernel.org Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page On Friday 11 February 2011 05:39:39 Zhong, Xin wrote:> Hi, > > Could you paste the output of sysrq+t here? Thanks!Sorry for the long delay. I've attached the dmesg output of sysrq-t from 2.6.38-rc3 and rc4 as the behavior differs between both versions. With rc3 it don't need any interaction. I start 'emerge sys-devel/libgcrypt', the tasks hangs and top shows a svn process eating 100% CPU. With rc4 the system hangs too but the CPU load stays low. When I try to cancel the emerge process (ctrl c) the CPU load goes up and top shows the svn process together with a flush- btrfs task, eating the CPU.
Chris Mason
2011-Feb-23 21:56 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Zhong, Xin''s message of 2011-02-23 02:27:05 -0500:> In the dmesg of rc4, I can see svn hang in shrink_dellalloc and there''s two flush-btrfs threads hang there too. > > Josef, it seems you are the expert in this area. Could you take a quick look? Thanks!Ok, it does look like the fluhs-btrfs threads are busy trying to flush things. Could you please do a btrfs-show and a btrfs fi df /xxx (where xxx is your mount point) and send the results here? -chris> > -----Original Message----- > From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] > Sent: Wednesday, February 23, 2011 6:27 AM > To: Zhong, Xin > Cc: Maria Wikstr\xc3\xb6m; linux-btrfs@vger.kernel.org > Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page > > On Friday 11 February 2011 05:39:39 Zhong, Xin wrote: > > Hi, > > > > Could you paste the output of sysrq+t here? Thanks! > > Sorry for the long delay. I''ve attached the dmesg output of sysrq-t from > 2.6.38-rc3 and rc4 as the behavior differs between both versions. With rc3 it don''t need any interaction. I start ''emerge sys-devel/libgcrypt'', the tasks hangs and top shows a svn process eating 100% CPU. With rc4 the system hangs too but the CPU load stays low. When I try to cancel the emerge process (ctrl > c) the CPU load goes up and top shows the svn process together with a flush- btrfs task, eating the CPU. > \xff\xf4\xe8\xba{.n\xc7+\x89\xb7\x9f\xae\x89\xad\x86+%\x8a\xcb\xff\xb1\xe9\xdd\xb6\xa5\x8aw\xff\xba{.n\xc7+\x89\xb7\xa5\x8a{\xb1\xfd\xbbk~\xcf\xe2\x9e\xd8^n\x87r\xa1\xf6\xa6z\xcb\x81\xebh\x99\xa8\xe8\xad\xda&\xa3\xfb\xe0z\xbf\xe4z\xb9\xde\x97\xfa+\x80\xca+zf\xa3\xa2\xb7h\x9a\x88\xa7~\x86\xad\x86\xdbi\xff\xff\xef\x81\xea\xff\x91\xea\xe7z_\xe8\xae\xe6j:+v\x89\xa8\xfe)\xdf\xa3\xf8m-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Johannes Hirte
2011-Feb-23 23:02 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Wednesday 23 February 2011 22:56:27 Chris Mason wrote:> Excerpts from Zhong, Xin''s message of 2011-02-23 02:27:05 -0500: > > In the dmesg of rc4, I can see svn hang in shrink_dellalloc and there''s > > two flush-btrfs threads hang there too. > > > > Josef, it seems you are the expert in this area. Could you take a quick > > look? Thanks! > > Ok, it does look like the fluhs-btrfs threads are busy trying to flush > things. > > Could you please do a btrfs-show and a btrfs fi df /xxx (where xxx is > your mount point) and send the results here? > > -chrisfailed to read /dev/sr0 Label: none uuid: 00eab15f-c4cf-4403-a529-9bc11fa50167 Total devices 1 FS bytes used 47.72GB devid 1 size 65.69GB used 65.69GB path /dev/sda2 Label: none uuid: c6f4e6e6-c4ba-4394-9e9c-bbc3d0b32793 Total devices 1 FS bytes used 9.48GB devid 1 size 20.01GB used 20.01GB path /dev/sda1 Btrfs v0.19-35-g1b444cd-dirty and btrfs fi df on / Data: total=15.49GB, used=8.35GB System, DUP: total=8.00MB, used=12.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=2.25GB, used=1.13GB /home Data: total=63.42GB, used=47.47GB System: total=4.00MB, used=16.00KB Metadata: total=2.27GB, used=251.34MB The bug is reproducable on both filesystems. regards, Johannes -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Maria Wikström
2011-Feb-24 14:51 UTC
RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin:> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >Haha, yes that''s very hard :) 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the process with ctrl+c and it disappear a few seconds later. There is no CPU usage. Reading works because I can start htop and watch "svn info" disappear, but everything writing to btrfs slows down to a crawl. It takes about 1 minute to log in. So I had to put the logs on an other partition using ext3 to get the output from sysrq+t. // Maria> -----Original Message----- > From: Maria Wikström [mailto:maria@ponstudios.se] > Sent: Friday, February 18, 2011 7:32 PM > To: Zhong, Xin > Cc: Johannes Hirte; linux-btrfs@vger.kernel.org > Subject: RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page > > > Seems like my reply got eaten by the lists spam filter, so I resend with attachment compressed. Should have thought of that :p > > > fre 2011-02-11 klockan 12:39 +0800 skrev Zhong, Xin: > > Hi, > > > > Could you paste the output of sysrq+t here? Thanks! > > Yes, it''s in the attachment. I tried latest btrfs from git (last commit Mon Feb 14 00:45:29 2011 +0000) but it hang so bad that I couldn''t get the output from sysrq+t to hit the disk. So the output is from vanilla > 2.6.37 > > // Maria > > > -----Original Message----- > > From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] > > Sent: Wednesday, February 02, 2011 7:35 AM > > To: Zhong, Xin > > Cc: Maria Wikström; linux-btrfs@vger.kernel.org > > Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the > > mmaped buffer of the same page > > > > On Friday 28 January 2011 04:53:24 Zhong, Xin wrote: > > > Could you describe the steps to recreate it? > > > It will be a great help for me to look further. Thanks! > > > > It''s a little strange. I have to systems with btrfs, both > > Gentoo-based. One is affected by this bug the other is not. On the > > affected system it is enough to do a ''emerge dev-libs/libgcrypt'' that > > should normaly compile and install libgcrypt. The emerge command is > > part of portage, the package management of Gentoo. > > The strace output looks similar to the one from Maria: > > >--=-sABADkyuScs9CgvROcG/ Content-Type: application/x-bzip-compressed-tar; name="2.6.38-rc6_sysrq-t.txt.tar.bz2" Content-Disposition: attachment; filename="2.6.38-rc6_sysrq-t.txt.tar.bz2" Content-Transfer-Encoding: base64 QlpoOTFBWSZTWQrCUbIArmR/hP/0AEBAC//1iRBcCv////AAAIAIYGZ+9Vz4G7c2WbWaQOYReLlR T7H0fKHABNgAoKAJE2nToaVEJKQlbG99my7p7wd3Unu2ZpXbMaWq11x0CkrMW+47liIbWLN5HS7F IjZmqm3Xc1TKgWx3Pvb2bMLItUxDZ7c5tiglw6d3HYOJPbXu5ve5Nlth4Qm7llVS2ZrerMoAOrOQ AC+7OoBPWhGqKU2mhRMhLNclfBmdUqGvOaKA1q92XrxzlshyO97V54KUpSqOoG4ds6YTOAHQAKSq iijtlCkiiigCqoEgoagmGh4pSSmJkAGQDExBoAZAYgIICiFPUg9TR6jRkaADQyAAap/koQQjU1NI MmnqAAMIAAABJqIgBBQlT9U8oHqAwgyADQABEkEBJiaAgQEKPSNT01H6p6j0xpNNT9UESQgDSEhS m/VQ/1UfqTRvUnpNGEAAaaP7v7f19bWU8sCBcn2r/GKfb0PpamXjbEmb3rASZzjEI4gSQzGhYSCi yIqKoyDCFMRQoUiIkRVSqaUUYoMKSBAFIEZEkAFkBYpESgQgi+WX8Afc/xPuxNTRBzpX/g/OjIm3 /g0gTMn+2On/E/Oof3RygeBn8YA1HZCHlE94XbmmRYRdusE+OwIWszo0Le7jgzAiEeElCCAKyc2L F77QLFVe0hHtRMxRP5u6Pfmjgk1uHU7A6Eq6JgDCB2IbJE5O++Gx6ROOAF4CdCLQe0SOb8oml/N9 7FyHJETwPdENx8owcILgxkrCVRwXEWQwjfGXqRk65U4gEPRnrSF9nJvbXBXMryBCNkH0i8YybeVc eobcAa7oZdX1YsWoNEiJqiaIl9r1ppViiDLScVqiJ2yYqeJCxN6StbnpDVjmdCY9ZzivdCqoYlrV JGSr7UHcN8gVjcFcZUW8M40bAWIM7Jujx2UVsAZnsAE0Bujtix48xF8B5fvsTgNoa1ZOlgDXkHjK p9rrggUCNgOi6Ap5A0gAbjPVaTtrbJ7c3z5612h76p3R2D2Ej3HbfFscei/VznJzz2RmAKOTjOyX A2R09AQD1Yk29X48BXkExADmAFoPindycjx0ehSCKigqwWLInHJ629lkBATbOs6jmeH4GaetDrJ0 H8gVsVhts2d9PxzkPnkULCzmEz042eQ+6KASSCKFggeaN7znjp5traQ62ExFx5QagCx0v3umiWAX n2GgSRMw8h1DQRYqwWALBVUhB35BXb1cyAsQOIIyK5htWcZQ05I8J3QI10EgEJQMXhaGRxL09xjj QWb4x3xg8zjHl5dUNjdkRViEhl2JI44BJwAegUkN95aMRUOepELdKob0qJDbkN8Hfeb2p2KdmZmi CUmLbJHKovgAE2LcgGAhJBHUuF0Kztt9inZhgh1lmbUCMPp6UFFI20VWUqrLQ86127AkzHgxk0YQ QrU8oMJjzOnHbqbsknoQI0496M8EZFIKElwJIMoABjJ8pIsUFURARgLFkBSCwRIqxVVVWKKosUEF QFgLIiLIs+NIqyKpFFVQFGKIqiQioiKn4olA3A/2pAqIkI94JQwnWk9EzFBFzEEBbJ/zNtKEAaIE k/3+a8Y79vHn7+fzYsft83Fa6e9a2I20RFqryzNN22cp6FYZmwqpWzYuazkYxmMJYiMO0xlqyHmE 1ZzczV1WHdNEYvNCxErNO1VjNWpnOcU2c1VTM5vOc2Lm7fMYbKVzdQ9YnAr/YaamrWda1rnS/WP8 wGCHTfeaf+fOu89E9z1seuaOJp/XFr+If7cj29Z7+l1A8Z62PGff3zgNzn3s8Se/QtvvvttgE+GR kigEkZIwkUCEHOcW+eONM/b7Wiau1nj8FYqm9a2xBIy3GLfCHTfiMAwkhvC04iIX1UOyV81ofhm6 zOkriPw5xi0hn8XauMe+Z4N16vOJ/cDoNTpeN4KnCWFymp6m9k6G8T1OozSiidBjE7Y3PEVnN+Le a8I5C8No5zOZ74d7jet7UBTGpykKeCg1MAan7TkMtPUdKNMXfvHfPj7Y5BGStZmaFuxJhAugWYUy IsQRCItVA0dD5iyQ0ZogS2S2FihbRdSnioF+yp+Ez3poHOVPTH+LHlcIEeiwukh2iQ+xjttiM5CL mILxCHV3qZtHIbz4TAjCbMA4+Yr3ddevtbmyqhBJBLYDFqxzqcYUrjhKbZw/fkTjqlvr34r2tVOH 3wH2iyWF6EIs6J1MmvO+NmK2QRaCVMPR5CTpdoCuIlEGONkveNDU2woNrjPnqSffvnfck+ejGwsB ZFFcgbuzexs4+HWhhCT+M4D0SxEBhCa2bOLh83b23sVkSEwoOwhEaKCbbhqzrQjEQx7OS1swkMNJ gm7ZjjNuF8vk4MQ3TmuXHD3TT2t29SbQxjwZJ3/BKmNKm72rpnopmiS3wSYYQsGDq0BbNPMFxJBG REiC81QYZTlosQgNZziX+QypFVBVVVVVVYwILuIiABegzc2WxyIiABF8LnlVWmIBDvFccli+mhFU rDS2JZ0w3tkCIQFixYsUWKLFLzS7hmF3QXl+8IaOdUhprwqtm9a7YL8T/z/0AkkQMIKmAvP74fkf 7fFqliWVX/j7YuEL6/8z/hBUiiAg/5IACd+faWQuMGQZAIH8DOMfpz7VjPxQFhBsERDIfmQdko92 0RI1oakd1hCMjlcmJiI2gSjJi2GbYAtbLTRk+kut5iQy1lyhS0EZdXBbUmUCKGkuwMlZCN7YciyL 8OS59Lls9t0DXa5ro8WZTD/VBUwKpzq1Nvt887r9pcrVyuF7qrpYECBAytXS1VVen4buFcL8y8qq qrlau61eF7LlYGlgQKvwXSwMrusBXsu6wMKr0vlcL0uFwqqqul9VgQMKiei+VgQPRYHSwFVe6+5c KqrA7ryquFVwvC9l7rAqwFVeV4WrAgKqqwFaqsCrCosCqrAgVYCsCAqrDypysDKq+9crhVVVelq7 rAgbrhYCuV4VVVEmVqwIGlVgZWqrhdKqqqqqrVgK5XCq5VVWrlWqtVWqr1VLRBUOglzSDiY8CCek USgAItEYQEov2sYuaWRajkgSCsRkEBjAIIG/42XIVWLZCXezrtod462AOdemuAJdIScAbeDFxfJa 0bMXa12zYmseuMTHTv5yhDcZ3zyREnkTaBE1WQJAgBKTVZFdtE/PWQJ7CTgSQNskSbiSBOwkyJM+ nLAVgQuu97BnmNhbYhh0TfONY1AgGt4JCwIIt84z8MWa5bYzJ1wgZxZTnUSBV3WFEK7WoGmknU6e NMWJDbIaWqtMeWlreXRTOWnHDS1ulXcSxTbiElutOMPHD7s6sTS7npe5ESeBJkSYEiJESIkRIiRL XbgpSIkwd6yBtkyUnIk8iSBNxNtGikAJg3rOxazAkziuMUYaTvFPWBUupT3iYGNBllnHWfWc3yIG 6CovJGVakCtXl6r6vAafBM910x1ojaMGcEgTBazoSQJAgBIEwJESQJAl5c4Z8Yk8iTwJMCSBKJAz zBWexOsYy4x8beymie180JjxJ1tmepM5Ze4zubdxJg9awOVmRIiRFPjSfXS6MNM2fG2W22mjSCzD htltKbUpSY3aSmyREiJtkyU2pSkyJACe4SaEjlpL7NJAnjY3Z6lCBDl6sttQpybm1KUgBORIiTgS USAERIiTcSUiRPFZctJRIiSiBOMNJgSAEvLrDIEyJESbiTQkRJgSIkRNt+cceMZMKAABWYvV1jNy U12eszNzl5z4qAAAGuGgEyTgTbk6KTAkAJ2EkCIkRI+jSUSPo8YZg3rKJPAkAIibcmSkyJORJhpM N74uaOG2W22y2201Taygy5bQUFKaQ0lUHOmvdY0w5nWNU5aTlpNWk+iTBEj2aSkSMEiJHlpKJETb uZKTuJtgwUmRIiRrSUSIkRJ6iSibZMlJjDSXlpLWmzmePBkxK7BloqrS++cQzMpNYODRczTECKFp SobjNYlIocNiTFSmeigyGmzezVFTTak20lGdGabXEDMGQdVmsApmrNY40q9mbpjOJjreUm15jSBs wzu49DhffPitAJqGISTNO5GgLJorTEST1SACUsRJGoCyqAgnoIiABRYUSWZUBplVQolIiVVQCkSA BJMiIgAVRACyRVqD9LKNCiBESEIAtoDSjCAMBoSCUJEKgDJRUh/yYWK2yMJIskFESCrAVQGmlqCq wFYMSNQAGkZCMBWArAVkBWAiQQiJESgoX9X7/9er+cP4sD8ArMv/sSgU/fACHXYS+BCH85opgVMI WBU1RXYBpIp/KsEA3E1CsbARCwnO2FFveHGpx+6t9RAZd0Q/jl1u3VaVSx0EKT2LXDcRdrbdEEA4 1sgaaQsGNMboH4KzC1somMq5OaF3N94iao7pcPCAN0S+ul1GFgSuDYyjYVcYlOAuhj38rm5SdudX ZVZIAvYK0ghz4DdDbTqxhFb8CWghYaETouKm6F03Q4il9DNjIi8KXiSG9AO+hO+1pC5tSK9Wwd84 dBMOYvCJGwKxOmjQckw1o2SwmcWLJYCy6BYCWDSwol9ERP7jR0zoYSaFi1jAi69HFtu9ZqjRXQFZ rYNerog/cTOIkI8dm/K1nXanVSArr+nfXQ7DQmMgbH1/GuNfYOmbBseZ5LH0B5NfpC/vVB16OgFq kOiInwFffnXXrjbe1vnNcy3vLWNgAJ+8RxoABqHJ+m2M+sTL+n6GQ4mnQMfAciUiJK/RECef6+XC Jb2AwBjGRFs5+Cv1AWDTGxCzX7Qty8O6FwDm5g33BWgVtkRf3tr98W3T9y+2JfDaHQ42b2Ex0yBy WQNuTIdc60q6IlkTEsI2IibIlkTkDME7wENmRAiJERW+kQB3uCaonCvHhp0BsZR4sPhvac99eAaB gBCB4QITtz0KuvWs7sBoccCaCW0AWgdIpz4TVN4CsXKYynmwmHlCrboGghBNYImxSi2IUg5NL78K lHE2wiYOQViK888ckL2vopNkKEzVBq8oAb9Spl1G8A8gAcrWQrQTqdOF1gFqiKZAgmyRLgbF7qLz GDDQyKJwbEhDIegshJ3pNHhTPR2ydAnqzayyyzoNwVgCy6FZ857pNdQGawTayUOiagMLl1MPAeWx p3qugt07UxpnW62E4oKA752DRLhIPVrh0Lc7cpRoQtsitVwZLodNEHvfugftspuQEEQ17krrz1Ze 8Oqb3FR1UrmZ84z5L85ZVOucoaFfGN8pTx1Px4NQEsvfFnFbVuiOBCss3jNZSUvIOa2gADeZCQBc QtAnAi2LO7vqGloCU1xRnWhoRZo0b7Va+AQ5K1uF2ArglMSbWu7EAuKZklRBEcvF9Bt9cd866ecL rCCkY0uRAqFXNfL71A1MnOsGG4zfc8DVdW4/OA45mK+SoXFaXfKTxeNcPl8Z1NPWtazXltLDGo5l VTjglIn85CYgEkMQP1/iErjnvGs+VLxtS9xiv+deckBP4kXr1Sqlj65+z59Z7u+U8yBeOvv7WxW5 Vvlj08CSAbR3Lh2IsUhDMSsVRoadwVgUEScda78sKoz3T5Dowjmsea4vvfs50rzzGq7OczXs8dLo lu+r3k6IpIZqtKK4rzTDhl1m9bzZ0PW9TUH+o0+hJpmBIP2goPRsKik9dFp19D2lQCsxKtKWPcvv fbk933v2KWUdxEZz6h809rl+cb8SSo7NhI81qIi2MHcREZie513tU4OK6VEaEtc4lRJJN7b4/FVJ ujbcNtw+XeQ28uzdHq7424gKahJYSRJ4g6ObbbZYdNsNkJJJJJElJJJJJJJX0vO2TQ5zm8M1SFpi JaC2ghaN4siE2meS53KmeNttMOpvCnQoriqEDqCbxxBKSaJCZekMdVOXfWp6YFMUmzA0ZaIawe9N 9t8Gpmq3lKt4GIu8doHEC0XR4QPSHKSGyGqFDSGrJJbJJaEikTgaIuqS2SQygGqGqGzA5QtmIMia RXeD5BDMW8Be4jv1NSrpY330BfIZx5Y5kg76854251Xq1WaDWyFQFyhiGGb4aIznT9otsMOmWHI2 CjnLfl/jAhIAIZdpO0VW0QXVEmkUEuDNaEoE2qjDEFxBRHIEE4ImqnMUBEGUlVZCKUCEOFPOfJiF QX0C17AwhewW0MS+6mHbcS+mYbZ29QcLXVPFNS0hU0NduCSCIZC3oYqvK5pco446ytbl3+fyf63x btFUogII2AgPiAIsAVIoIGS2LKruATQzRGY5tza1rH2OnYNUXYQGkIqJkJikF8QsabtgCLp7djXG 2Zrzpzl2IavTTzFhcM7cyScOco5Q8lQRYbGM6NwQ0QRTqApmBvcQGgCEJcNZYBAlhxaxcjNhNzbv fBJkdVRLFKWqRVVYIKsVWLGpIDnNwCTlO8mpwiu2mgKb8G5ZVMkETaUwVUN3bkVebcePTRC8hEhJ NogdOusqrCoUTESkiHOdCwFCLNd6FO+ltdr47c1e5QbPRvmmqae9JqbxiMVIsAQBRESMRBYoiRRY sQFEjFBWKsVSJFEiMznU7kZjLp6z7rnn2THJ0QFd9zctbmXCjtgeTPlURE1gMiCKhTgRGKG+tBsG kdOg7NFVXNUvtlK0akM4UURFeq506NOoob8aU1UwqnUBbTANguBYM3XnTHnrVdIbcFDCAVERwTSS AO9a1KhVVs3hEPowREACRDHeUPacCp53ksUcAe/d8FgyIu9ArqJzzEFd4KDxHMJA0AXhoC4Eaho7 WHJcPdUQhBQ2BWqS5BryS9PA3SE1j0Uqq6vCMRiKs3MAr1RvipVMvQAaQRsKkXgTqgK4ygUeT4Qe X3rLMzM0ZDyUjaktTgXTDoIQSvO046df+aAgHxBU+Ag0KyEFAgCwJESRCBAIiLAkVWMAWIRiRAjG CoQIQFgrEgiyKqBAFYpFViBCEijECADEYsIIAkgosGEGSRICQSDFBiKRFYQUEIIMUBhBEkVisUYK sWBCDCAyKSIASAKkEBiECAMVWAAQiLAYEEUgKkBiECJBUIhIjEiqyJBIrAZCKkQkVQiJBFJCKrAW RjFSQRWQjBIMEZBgKBEIMhAYMGICCsFiRGALAGAsVEiAkgkREgIkSIokkAggEZFjAWIiCoqMYRUk RJJFEEQWBJEJGAkFkFRUIwYsYEZCAwFYosIMJBCRSIkiSKRUBgRgkjBAEFGQEYJBgiQhBQIgwgKk UCCr/tBvsqDopBHpD6KQaQgoFHZXZ8h36L2AWALEIyBvX1OhPjPHTvtrOGudMWvb7T7dZ5JmHv6+ e09dso10KHEuJ9QUOJbeA6okMMEcQQvFIAvR6nYTJ7luMEUaEA3Syi0wuCv31wCvYn51N5ITslnm EUAKMIlEkn6v8/w6mwapvLq9FPtVirKLGtHE3NFyfFWknW0nYvyBC++MsAPa0CyCp5wxFBRDIGb5 86tVphFAt9KDjUphvSohuwDcTLSIepEpCCQIKMyBQj5iIgAdkM2AFe5F0IFokprOpTcq1Cga59/N vZoivRuGoHEsEQktWwEwSDJpWC0mtgfKstNREEDVlSZZ8kjCQ7ogTtxpd1d3V6yvslARM3zzgUXk 5EPCCIluNaJO6HPemrgLMgCGUaoQVsNUkQigVVAB1iHMyuGhrowUaTKJcBeedd4YwDJoDoX3VcYb uBAtyXsr3gXihPTdqDC+ghYJtTkREAC4O6AFg7m6IA6rnmwCy/EtqX05NzQVCoHTWxRUmxa5oqCh bQpENgVljnNwLQkO2p6QJkzvtu4iDt08othH43oO3kIGELKr2YddiEwOgFd9OOPBsigFRdUBRYIA dpqKt4XQ3VLna22W+BEicaFpj18cBoiuONbcX5U4t3MFuQDwWyppSBvqnJZ77VVV08zZhiDHg4wG cMUUISQ0idb1OXggqY8nAF8ZM63vWji4jDCWWFH1DKH3BRo0niW/gkTEYe3yNwt8/EbK2cPl5e9g OOst5Inp40iJ0GoIaJshk8XxcAXkeNoQhNMXLXJkyMeZZtpxVVyIEqENc2x6ArpyHaIBcypESIcE PEVVWiTuhXSjeyheIQMwh+RCO+5KeM7mzkMcQk0Bh9ecV3s7IbIc9gbokETo36DeW8vz1fZRT1ji STKGhgLnbpOOjwSQjCkTJvXJdwSYM6LSA3YADgAjk34LgCIi2mIh1vgkgEEu7wHDsyd3gKqodQzB AEZIIvEwnkgBQO8WCCN7fh6dqaHO2MpNTg8i4HkcpnS6WSyANfD7efbbrp016YrNqnz7dp6Fenr2 2r0E8WKRmTngskwSnHxUAzFEISlLIEYp1oCodiFQVYKorvRSqxRQVTcQCldV9INQC0P9OuiJ7eDp 4RF6GLCazoeiIk5vNLoL7gRiQFYNrkhJGYv31xDFEVqqVd/fwR5zQ4yV+Scm3X7frlZ0MJNM3MMM R7wHee6lVsPIEE+xs+uhUpuX3QBDXMKBemS+yuog2YKjqQECiiMVKES1AGSwG9kADr6l0TQsx4NA S4aOllAeOKPPbYzo08akJpzczrT84BORqHrjJIyTQ3Dt2RLPwxoSX5itq6xlEOuVdUxnkNd8DbNJ kVe5zZub9vK5qCkcIa67HjsIiABctOm4hO+OZibknWvA2a1GYx6qVWDNCLwtn1VJYPDyz8qyN+TN hVp0SbhRbVCnnWj2CsfL0V6lPYbDwAu0+c8/UNDZM6qU9euJKuistarCrcgmvNns2JGBOBAcI3Zv BFaYCuHBJewK3XRCmMYFiBrb4ERAAuarbJyhZc3MBh0GxL2CiOxxyNXud++oHzE9Tf4HEK6OOYqq vXFcAmBRqJTnHXFdn0gSEjmiHAhiFxLzUDlxsborbfgqsUbKoGtG6vaTSfPKU62GUCV6lQCAQTXO dxx1TwGh3LOYZDv2ewS4onfpjqmXfwCtwN/uYRLGNPXaJ6GOunnyVUldVuhZoKKnzgfiB+T7qSHB tsMfIYDQj08GzoWC9h2xxqONLdAV0+uM6sFEwYe34k8/JTSqr2Tf4XB9cYiEd5IVWfdkUq1iylRQ ut9DrtTnnIDYVdC6nXji6nMQhoBJPf03bgAd8a01TXsvTc/KqJWtFrk7wBkYATUgA2bcnpD8AdfO ffylNdukTYC7n1n3V1u+4CvvkCxrYpahjAC73VLboXiqezxjGNOxWONc1n2vr9MWxz8fRkTGSCBB 118Q+7xUNpCFZNYDIkdcjqwRnZWzscJy8+okee/iV7PMhYuEZ23ve56iFQ1LtVVVp0EzqBEJqOlk klN1xDb9U5357lqsXSaqsgbqrLEI5hbu3GBA3A8TvU3x95/DSjCm/LYAje4huv1M55KtaVVvd0ep J4qmNIr1cWbkhyHI1yDi9ywQ+AWWnfmiymtOnNHWvNbXe0R36pBsERY7AlsEhSIHOcX7tc0smQVK 4AuJiqYm5EaYCakKSAwSR11lPzb6LODvd557zKq1ap3sBGXrX5zs5op1QrY4YMzIm/NTVBM9XF25 IzWiobHORZRVwIFa9b7FXbVWngqiOpseIYMC0dc1OU28x57gO5NEOTG4e4p350W0BXcKs1oX8Phj swHzjrGVVVX1ry378v1A1JmSodJrD/rwD3725G9IrxtfTNd5tJKhgQw7JjHIWVXvbXTwoE4AYnp3 uTHwDkREADSA7Gvs9qbbiamRV9dXA3+hUwd7e+c7nJYLiaaG6fl9/BVvEAO+rXTfJyI+wiJEV+Cf Y733JRKAPnzt6TBE+rDSvKp8x4LwI3RIC0FXm1pKNn583fIhvuYkh7jp7WqaSmuqtsp63m+vzN/w 5gTXwB8rnkdJxbTvCqq2cqlb+mEuhcACdaARD5XuwLcENvNsccbm/aKJt0V0A+8+c7FU6IIZ39gN ABRBKID5dUCyppJ5K33rY09diJA7r1VVVVoo5Mmtc87l/NeAS8eeEDHeDhxL9+DTVKpnSvRAJihi cWowg+MaXt0dm+R8ns84hI3uBJSgZvj1iEVVIqiigRrjqttrCAe71oDE5/brGefOOMctPgyM/F9F vXae7vBtY7naFWp6Eha9AOCdJjnoe6Ap332ztjPOzfIiIAEpyb5KA14xld+i3fG49m0UwqW0rJbA w67Z6m3HPrXMmgneagW3DvzznODHZtn/EpvkC3QqGoZO0LiFApzBUtuYN+xF2dp1a9VV7wqIMKjd l0FTjm1jmCB7sV33rzhmAA7wTfycK64u9LlWtAcMgeIBAYhAFAijeL1FU3iHfHmv3HFj5nfi76+o UTemrlArXDZqyXsBaxkloipYSCBT0bYzjPdG5ocsheiiR21uVVY0KygU1a0ABAgC1SWKjKAAj8V2 tLmC1zexdsiEHfW5w6YJwSmXoVa0KtGhFoLSWZa9WCXSmuNJKKqwXY6ArQwRttoadzR6sU5ooncp hCdHPHonzRpWH1G7kuUs+Xv74rUWOcAFY91QZgh1dxPbRrDvr3iWMQLqczrnYoLn3pHDQtZzSlUU O2pay/C1Us+Sra2be2aJ4BBDi1CYSCcJhEpQsxUxBEpIAZgmsEsxA+V2qbc3USEBZFjELccE3ANU rjmrLIIcHJUKYdwgzHPS0VWJMuRJloSRb41LIXMhGBwCtixZ8U05GigjyEAZwXtqUH3eEk5fx+FR VXvrf6cepL4mhqzJKjMXoYidg46BYNuNdLqHMMRMldmDXrdubHYooi7k6ENdc+qqrxNkhJPqQldF QK5AoEgxigighE1qFEdotMYwVjGME2RgJRvRmrJZgdlr3W8SMAkBjAPDYRcCnr6ympL83S2bHB75 RE4XTPCPYRxVDCI7wd9fNiqsqrwBD4ZEDaBsbQIwO+kdjbfw457+KiquM9e9ZvZncIHragA9DLYM os1yVUWIrodkx72JCB3o8NNJYawsBrfybWSaBtAM+7Kr3voBuWniaHvMCydzdVN+2pj4gj1xzVVV VYV4U1xixdDpB9sUFgRallh38oD2QD8bfPEKChF5ScaS4EVubg/LaCa8YyYArWRoTbXEsk1VH75m mxj5wHbX16zf319kF8tq1vjOR5XFat2yOgZNQ+lZs2sUIb39kAObIro5ynng0Peiv6n6qXF4o1bR VVTpTIErXZ5mvuTOh72gVAGz3+cc+SoQC7RFBFEFxjy3XRKzg/huG4K7HTk+d8anmnrroL3oDCdB 1rnPBKJWUFYah11zgVMcGYWuXlUm86DoEecob98Yc88JYVU7xFUyiJ2shrghYk1+fJ8UGIo9Vk5T wEBrGrd2L53bWEnmed1NANAC7zSM8ed/bZkStQNAVTPYULqhFe4KeRBXWHHKJDNkDvbfUtVqlWuH QK1r6LC2eC6bkTZzDrYVVVdfL/fBClFSmUeVuVr62NjRVXoFbNgTJCBCgK+UyPhHMfXspczs92JI whOhG5FUImAgazgqnRCER1dXq70BE+HJvrcL7GALnzZmyF153v7bc5yd8m/dkTBupkFdkeuPYaga ARz66M3TvUCncKA+eEPWdCqkpzGRhETw4FVR8wY5u4+g+aGouuJoTSHmlzsAmdAO4KKqqtyHVmYb B8qJRUEZDuU1G8Dq9Pgi7IbvDCMJJ1g0Nqfg4YIMx42wYMwZQMVru4VEajuQvjec1dNgvyB5Cnc4 3q21ohggGHLBckMaCFW47gCOSCwfljjbBUACfMeScgjKCR2tjWu27wk207gc8ViuHo9jEVl8kkkD JwqjBeotqiTyD24SIQ2HbBsBvds5ySaYe1hbjg45pEvQWCKHM9lh04EjkGMCKmQATmB9pmeZt98s qr7Xa+0P73r5v3cxwsJHxAmozyw9ciSgiR6NoSyas7zA0jYn2XrdCUmLKESAiqKMgQkhAk/P2FEp +j1rU5tSCAeWyltplykrrF72Tom9jhVZUIR129c4OzkqSSSJXzz3XlAUXc69OcmyN+ghnQSlM8++ ACekL9796qrWq1SSt3jW19kRN0FbdRrWrRIkNyAMABn4+Vrgj7LWrw7s7uz87E7FHhJGOtIrnYTq xgk8DJtfY5/O3QmAVgHpBYRACPcQPcA9ir3Q2IoOkQC8AHz1nTENKrKoHr76m2B/APXe4kON/dSi 4puV59zstqk1+vW5ok4ONktMJmvoQtxyUnBwAp3ERIghVfNLfNDfOCVVquy9+RrUTibCijKgMoW+ Xg22Ic8myqqr7xCrERsqke8geceeGpAJ2Yxc9nokCiDyBzL70bu7mSSDASBaBpBCzFXVSbIF+zUm 3HMD2NwFv5gQ1PWBRIJz69BHI9UIm1jJ5AHXnYVmGUd+YISTXyubaqyrugkibCqI7hRQqd8d+e2Y R2OKqlVVo9QvGpQSTbsmGQANVU9YpAxeqAuAKcxFKUrIEXmJz8yJQ+0mhpwBRsXN41OxVrno3dMW SwC5v5oeXs3dyqu5xJsLFZfODECAGx4XjSZFo7sEuaavRsgp3whylkRKOZOrWaKcHAGhbDYgVQA0 UIsCqkX4rXGN+K5vyKe9bWLfdjN7c9eLboo3TycXt3I9sqlrWExM1EjFAqmyF2+yEIABRsByQzAw UFwJ957DWQk8pVErjuELwehpVVX1j1ZTPG7d5jRpWTou1M4OxrtgrPUs8CL2AXfrNc0chGNmsqrg /QKnlvePaZm3iCB6iYEBJL3tzOjWO3abzSLRemRNESLRKpoM5UqXobmoViGEDzYqZ77apqqJnZPg tUFR1IJQjxpXnWcFNsCTpnzogwEiWdew3zzqnSDyiZKBAsRGP1r5zpU/hbc253Tn1RcI+HN3sdec ZolV8Cno6KrgOL6Hy7zgMHR4ZhA7Y2AAhNS5Wzxw0L505GkILw532I5sVY5KM6VMIupoU/iHBlFV Xf8aVUvCiijU0MUSHo59p6O9NCUBNQuHuCUwjI7zwNeAAgU03esh3ciRFEDGxUmwns1BXzIq7G3P rQOaPSGu+x27beg461U0CAd6zzji+Ns61V23d4h+E5NePXZsiqrcAE8+P78a4HXYI5Ix8YtbJLPp DWPT7WoEdTEgkniqHdXd2qDdbwsPN9sosW8ZUsORz3YQrm2oAk2IcqHeWcjjWRGxEQI5nnjr4LkO QOGjjlt1mgCSkiOYFB2DendwNARy+9aien3PJGwBK0cvQ63rZWea7rRa4zvWoWo93dd7vu+85zp6 T1JE8znOc9xg7iIjvNc2+cbSJZ6kkkkl1ttJJJJJJJNttJJJJIklpJJJJNtttttttkkkkkkkkkkk kkk+9FojHmNMIX0mh9Wu+0vKHxUoR17zck5/L8is3kBmifmTxbbx4tbYgfPAAc4GLkvg04gtg8bm uY61jOleuIvLD21vjHDbfpO65+M4mm7aO651pRluc276Wt6EfTHi5tqgUttXWW56rmd7asKn3G8t 2DWnOVR0GLyM+zVZi9xmdgNVxEWPZ+avGj5dKydLvF5z1ha5i1tbeoHM+by9LB3XNb+NUuNJ18vG uWp1+GuvWLqg9mOzhl4z3THRqB4lO7TB7o7eytYln1pXnotPvSmwlg80h1pkSyW80NWGrcuReORQ 1yBoEY5MNudOkvEySyzJdM61DgeVLzIzoMKYCKEdBYyw3wOWYoX9gBMduJBDmtLKKe7aZTjxmgK7 2sdm+3MfOuPK9+3adAa9+KuqFQhm/7dCBbSE8UQUhFGRQVRjEEaQOa2RGIomTp56OeMnvbb5UKbm 9F0O4Q8NBLgXULEA+8/cG5uJrw6aO8hEn37lVQ0Oj9kgfA6MkNmG3v2FzOmhSCthUovs982dd+gg me55D8aREm9CFMKlLR1NTpWL0eydUHM5NgNkHXcPfXG1s+dvqJqaPR9IrKx0Bo7/hRVVfp8l2LEE +HGhDzbj0Bx7FGgs19e5vtOeNTMd+DA0iinyd/xxqBPh7NUc8WD4hchxj1R8m4aomiWBX57UrMSK S2QzcAge3IIfrLj3vvkjTUL3pK0VxeUiqq9XoSs6qjyepNvpAJXBvtizeEk7upA8QJNvtac9Aos+ amA79+lVURWE8upd9jVJQeoPNaIXJmrlgMCHXo7awru6yQL9uoHpjpBDc9o2DFts+9RLkYNvT/ee PLqOuRH1z4/3fOe8bee+on5w3we8clzGt6dwvbwOESfDsVcgal5ETdoFdbmbqiDnaSFDJqhd4VVU XGN05fMcVWm3Il9mXQZcs1sXIcWsiBIJxgyjHittPbnv72zjoUU2k34sqXeyIqMRZ3U41RCpVFR2 te1DqGul29Q8Xc04S7BNSA0QUhuJSkEjIo+RZACoF2MPZttp2ibbgeadZkh0JR08bol7MHTU45xz CHeTuWcmChFrqKpZJufPPNERYiiOYTsPB/NGMdGTf0QahhKcca2TZPuxTymE97+8Xe+4MCFNansT Ca90B3CRgj6N1T1eBQRHEJ0bY7zQqIKGm7nGS+r0FVXgjQqIqKle5IUWI7hDyrVqjobEb9YshYED udeseQ433nYGwaAXxJB1k46OwOwJqZNcWKKCKm1g8WLVl2I5MQuxUERYoqI1GlHSFUI4ngSS7Vtp VoN6uPRVKtYq1Z0BzdqqfIw2eYQ2sQuft6AnhPfth0BoXGTlaqeMharye1TzU1FFV+mhDVWWVSrx rg0Fxke6oR2+6SxjOxMir8aVeCj3j7fQ6LVHDD7jTXXTTIik2vTONta1R2nhdqvc7297CosRT175 KWcpO4cOEF4NZaISfHmZPz3MRYQQGb+PQDOV45+4wIqqzUqlXXnWHuejvZ3u3XbdU0CqeZCC4rc+ B6oJI8CPMVCmggu7dwl0Dg0vchMcBMHw7tOXtBU/iIK/UAVBXX+nwqqmREQAI1ZREACqryw/+1EB B+nUTxJkTzAt3La0eMTzV5yh6oOqB0VuD2U1bNALHuuiehRruJcVMQ0VT6UUDsB+yGXlsQtncBC3 FOKza7Jl7Ya724xkRfA1U0NOBE4LBzxkwnV0cbbwTI4TnJg1yamtQ7sKu7oByddeeBc47RNL5ztg 255DDuuEroTNaeHGI4dUAMAQDnsCgyZ4A2U7A8ztXUL7Wv1xdpDvYUTdDu1zhR9W9V6MmcdF+spZ Vz3oHYqeWVOEVrsE7AZU4Q84CzoiuETYgC7Ybar4WNToQ225Huxr0iW7Ob84u6aJvbpdM61kznfr AbiLvqnPpFe+U27Ji7cyAurkUwFpt11fjpFeOSBDHIC8aiGETxG3GOCQ7Njvs4PDTYoDgwhoCvYm 3hDfEFErzhESuE4A5ApSwgeBhfEwA2EgXBWsIdow3E7AueciA27BW6bI0bAr4CvIK0IbInoAqoH9 Ygqf0EFf7RUUFfT/X+f1/z/zf5/0/jdpMZ47XfLf6jvjf8UYQhISE4gxcff0s1kHKM2JYM5JJzp0 51aIdDFgTyBgU5CgiP81EBB/+hKRUD2/mkA/kJ/SyOP9At/kWxSE/Q0/xUZe+v/57iaBX8z/B/oI D/T+/k4EOFOjRwWTnH/BE0U6THiBkVOB4O+DAcn79jdwKvISm2ToHdKyTsSFV/k6of8gNxPP9Z9F j2IDwAueNhMBXZez4MoMAnOrAVrX6PhcDy6poBsohnvottrndPYxejw+jZ8TTZXmdoIYQQvxQCnJ 0h088hgDGcBgEvjSawUTV9DnRB2N9s8ON00BXlFdEFYCtArNVBQ2E/o56/sfhr6Q8DJJNT98cnW6 BrRSumQuCE08h3RrAsQDlB8gKf4VvvuH26R9XROTUTg+5g+WQ1F96AlZ4K+qDh7FWIIaq7iac63E wT/ANxC/YX5ShV7nIC1WO+qJshgIj2iIfKgKa/G3kJ1AuBJZuBiHMIwSTIHFKmP3y58mOe1e6ACY DwEwKQbDkBdr6KnRS4nvyeCG/RuluM8bJyO75Rym6jmg4uflQFPsATwdr0Hwdw3XRP+/PQA6W7lh fLg/vsZRXU+1jCpS7t0U+YinWrNIKdfY8wHURN1HKj2NE393jQFaUXQ9bu9ARBcBpksoKFeAd92O UUTPf32RNRDsCsimgEBWwK7mVOwTBZuit0NQF0AXACFwClF+ILkDzAvf5bJYE5TbZPD0dFiNm+9/ B/x6Nfv8Pv+ET+lmQV/rt+uPAKAlmmof6vl4u/7+P6/4/1sDrfPLd0dQANkSmKh/XBfuGO/7Irpk J4psp2SEJgtVo8HuES7VGqpXlP5etj3x/iKKz4S9TRCQ7m9UTOvRQK+nBrLIrs+KccREiUCvRrog ht5qicyvEsYPYQ3EtRnskJD1ZDfyUhe2bNut7a9dgV1cjhVu/BL2Nzk72+FKivruGE9KmTK98cSH TDN/DfqF7NIlUcAbr1ni3UhNSwC+reHec87hCGtQzUKh2FauQ5AXnBZEsNUisrWxDXadcPXsK2VV eqoYhjfjgOBYi54DgsXJPVu+NnCIXClNomqvBp6x3nS10TYBa3RsbI0CvIh+vofWPXzx49jFp5ri q8/Mr8qrwavAC4S6Jn9PYtzDMmtUVUhI9xUTAfV8JQqip66FFEqWLj/B/KH8u2KKmE9N0T+n5P4/ VgbpkVfHlqedkRB8zsVzRDulD+ZdUS5sHmni7KJJyn4Q1PbbToQDIH6/WodaGMhCHr0OylPZxuy7 80qJYn4prJQ4QA7AdLL42afdRU80/U/IREACvB+Sp+wionr+/7uhbswtsQgYz/ogqYAQP0dxIE/J +hBEQANirmHbIvekE4yGqrgNWhtb9/6+Qu/R/HTQ3tc+Ir2hsoXROeg7UamC7gSCASKkRizc7nmN c4+Eq6VLUG4m9GSIPIvKLtnwkTcTe6IMQQsoCmqfJ7z5/d/BZPvb6Nz0XTJ2R0Pxi2fP7aFjcBZp tNW5xiwkdH9a8/ovwaH56v2BeiwiIAFZkLP8vGhs+ArCkTooOvNlfAKdvNR2MyQVjGERVVFU10ap tgobf5HeB+cc6+jYU5ooer9kLMC+FAxMc3vt1D5ryTq/ESoAe6xuEIcgr3wCuxfqHoS6q67OCVwq nSRR7TvkJx1w3JGEhxrTYkwbwkNPDQOdjnxETU4BWsv2zp5sgevZ1nzsIQyaoGt727JpjS+PedEW Dc6rejPlpBVV9TSi5ZBoROLJ7nctc6l7Y7iKoi++ali6+RQ1vQh3n+o2J6JrRS6zYD5AebHoROkm 3HgDhE21GjwBePewK/5/z7plJ7/wNvbAHl5m84z0h2Kz1JCQyiGqPUFdgxyj7gbGm5xD9/oK/PyK Y551FWxSSUg9e5+ibbQ1MhlUVVdtW/V9+HlmY+6ZD28T3Qzn5Daq16Z0hW3cznua50vL06J6nmot w7NBQRBHVIcOFz+HJI+ej8SMXxlKmdqiHoVUCNU7tYAJHyJmJbyNkEd4Fm5A7+g+pA+r/I2bwJ9v ftf+EQDHmEEvE8+/bHsnkdpVq8y1zPqpVSHndCkNtD4k1XCm0RIevsnwb4fWxoVqtL9UUcWMOhHX 393+TCXGMRho4SKQQCfN+3aHDsiaGRC7FlY0XDiZQMmKDatOPMbmMYd86i+c91pb28PjO9DULUfH 3Xe77vvOc6ezJ6kuSlk05ERAjuMHUREd5qXN5fONq5LPUkkkkh1ttLCSSslRJJNhuGklZJJWJJ8a SVkklZttttvQbbZJJJJJJJJJJJJJMX8Hwjf55CcvikBW5xKmJDMSABEpGQv2LJvkkY+pvOhpD5+R UCAkkrRQ9d+cOImXXzHUQklLNkN6GDCWL8c3XAA3qeutP0wFZ+RsXxjRLQC/OtoSH+6Cp8RIxUYB FEjEEgRFYQRYECIkRIwRkiJEQYQYERAFISKxgQEYowRgqjIioMixgBIKSLAGEWQWSDGAAxIIkiCC pFBkREkURZAJJEZFSMFSRIwCIRGLBIEBFjEBGQFUZGArIhCIwUCemTCWIKEgRAgCxAiqRYgCEEiJ IgAwIABfsgYNNMI0UmCBb8Ph9QknNFa1A+l3vFU1LyeV7VeCihU+zND7rTcz7DHi4wt/O05+nszN 3iBPDS9iBYCICNejmIm4STVwY433qOSjw0PZuu4G82Qu6HnIIHVxQI7/B9QUkg1mrguEkEoz0Ml4 axWGvM9iqqr5ekiYxFUY1x7fhKa3rKV8N0Pi2unByUmql6S6G3Gxa0IQNNPeqWVNbm10qMxyWaXK b3vkT1baydANhLcm+sEB9PaR9mhKQIEWQKSJ6gdcgbTtE67khIfKWcO54LiA/vD9IWkAddz9hRIq giLFAX3xDi5A6AbDZz4O9w6+mNzK1UkkkmDt6ZLPoTes8rsBdsO/79D4+DfqeXUhCfFcd7999SFY e4dXTJ59DqGewfD8EFb7ipnxoc3fLZBUzNUJjHoUU04mhD8/uECc9B19EWIq/N54fmzsfC+6+aWF X4QAYp2/gXV6qo94IqGwq00CFhijIGv41aFCbBpYECwiRYKBaBroH0EwNw+sRKfzCtBF1H1SKa9B BIwYKByIimxkRP1N0ANAPHt7Db29ixaUHz9H73vEsSnt76c/fv6f9Mnzr2/jsa78Ty/K/486Nz71 +H1gefW6fzI+UBSHSlFPvBc/QpBdEMKtH8M1pPvFEiOaUdy9KKbwXaAY0qxHAGn7CLYz2H8hnsqc hoh3+ouTe/1GEhCdtAVtY48irXlKqqq44qz9oyPM/fTbbvcwd7RztwJpDeBmdTTd0Ny/HW9CaREX 3ipwTGLSSThC52K9VQiidiFd3zuBNtz9s6Pmqu+rF7HrfKj6Mce9jInIWYAABE9i6EBTE5y7+MfC 9S2ayaawHfm/sDzrjC42ZKQQCqNjby5Ugu1ShRjRCTPDxsBMHyBfsOd9Rx744VVVe+PlmBexpVe8 knHLzkZ7x7vOLvX23rg11seiHfA7AhlgLJ8FXg2Nqv1Jcgghrh1MF9zxVxKepqLhRDNUNkeiAOV2 yQJXt5bnXbjrAeZyis1bp0pENDo/Ae0/RQel1RPsWbS5ulZQCfEpFfzUaLKXQum3r0gV2ccl/X06 +EfT0AXBna25+oAFTlUTU/LkP0+qfeIP1F6gQiJ9XwdaojCMr0CJ0SELfaOhCF/ufZnSfX9s/r8u yqqr8ODnFzY7mX+F+dgTJoj/FMBVtbDV5pnzAd3sO8v1bM0RTqWaBVtcw8K2t7hVcvXGQd/kV6wE EEPsLCRQS4jMVZtnnjnYJw2hQu/mul4334v1ybIfT6X175fagfE8gQE1OTsJj6V1NvjqAcbef0EN gwkMxCwlqYd/ccFwEPzC/zuARCJHvs7486+329AUJhK+DNRjER9fr9XhheZicTzmReJvQI3yeH5R A52I74JH3IHg9qRPBAT9SLcQHIgpAFiThlFbZxoKpPMf4wSRPJAjnugfQ+lxLQhJAYidqTz5PPT1 320C9WsJDe4jRKT8p+P3x0Pqe/bzhrcADpyXPj9hEQAPr8ip1P3QQD9qIFAMUax+kV+dptsAsU/l 1ETwhSpvr+bXLUuGkOaS+13koBd8afNTkjqpxuXFb0AB7Ce9/n4RsIvhgKz8vYCvSLAwnVBx7SQ6 3EThLXCM1v8W03zab0VJJJIH1PnBgFfy6bWIuOoqUa3Q109eEOT1n16wLxRQaqjvCF2Lf97w0s1F rNK6l+Ds/H6oMwQsZLmANsDJQnMTmKSz56nUDwR6SG3kTCXI57OPLAkbuJ3tWHfgdkAAdiuNbXtK 0PYvVFRrjhU3nokwBvi2Z41NheJUnXR0H8A6q/Byyji9MLflU1+jbXHAaykkOpm/MohA61mnfZqq oreXMxMKMT1jbjGPQSO2mOei7xwiadmqNjxVMw9EISStBMEOLmCb9PDlBU7UoBelU9jpud89nCjy Ca7TUucQoFbHj7+wfb2JVEUu/LBPsSzYxYsKsRy0gh9QFPshqKZc9NS5cgFzpqBkRujYD9QJhU/t Au40Mfg2VMCJQKwOEUTYDIiHQQPsCvn9PH1Pen4q1/ufPv3mnv3REHxBFK4KPMCZz/8ERAA/Km/r vHlBCH4Aye/JD2HR+oa3dy8kkJJz6fkfg++OR8ug0T8qKVjmHsKv19gP7Idz3HseR5aoBc9vH58j bpTIfQkYSap90AiK/1Jt9MgfPxqbEnO+YaftyKpbKndB43+3172foRrtj5u2AWtgX5u67ZwfdVw4 gWFUPlmoGds+j7V8Y6TkkJD2mxy5XYQ+0N/wGC59dQklkz/MJsIfx+U5aERAAqAKRAwfAGyCHuoj b2+HAAvtoJfpgyqhNURPzOgbR/khQH7hbZE/MN4CuBAbocltOAFOHH7OygGoZ87k8EiGfWjduJQq lHSvSyD+yDFH+JqIDRxHdwCFaikRLmp+yJgUopEoUupkNhNtL0+y4BuIaqACAYdQS4aODMTur2A3 H1f4RbBcEiqB4siDgfA4wgCHg79huFQFCEAiJKDy7iJgh2wmd1ByBkD0O4AFrABonCq3FwSFhcB5 gPy/sT8H5ifgPhSwFhPw5gmIl4+wFWAt+FLfrL3LVRt+dgV/hC+JBLIGywZHQTIpHm9cJ6vMaFi6 /MXG3/iEzGVvCiGkQ6IgfsZAXbyVTT79QQDkd7IE9H2wRhebY+pJOGLziwmLTnD+ifQ01IaaEMbt nq2HA8DAEmFm6Vw8ckDkbGhSMigluXcIKBJrAh2vWsFqDJmQQQ6BzffJBEazdYxwBPU10IuOeYWS EdjkEgi7pcUH6TblhNQ0PzD0nZ+NTUpmV4HH0FZKSQ04MoQOrIt6t2dM0MMpszLis1Ilnc5J5S7I HWbFGzogqQEYKpCRSEgyQgCpFUWEUkQiQBQiyRQgkGBIiACxYCxYskWLJBIrFkhBJGRZBSALJFix QFJEQIKRZCLJARhEYIIQEYIVDibStuOatiYo4FQDuRATMFBekRFe/VIXvrVQksgAj49EnPHc7pLm CQrQCEnBZ3jnm8TUWzCjPFXUoup87mHf4fNq6TB36r0K1s3WxvYEnokIPfJsfNQOgDdhXTttgoWX nTNUzuZHRBGQEEEUCUFIJGa771WjvsO9C6kIekhrgUqFUb0lbHs420h5MB8gRtFVV4qoqqvXcqHP towPbwRuXENPXs3uKbZ3zexXlqIesYtREDJL2Eo76PAVp0dziSHp5cHsQ2QQCcdes8rugByBuGm5 6Veg4bGmD0BsB2Hm+3rZ9mt8avLf7qmjmgRIAYydbUWaD4jzwCJ1pOtvUIfDG8l14WvcHfnlFVV1 +m0mezqB8uWbbGmNkr4+tPNKa9eR6rOoerxi0L7CSytggitzlMHBLDGOmDYoUDvBJcaQRAP359+Y nVs3a4hppvYLCtpchNjrJva2MXegcONkpbm8OvxcBo8xP7pvET8HsFSBIJ+34KAtIKeD6iIgAUJ9 YAXUkgiGYoh+Sr+gK/aAFNNGB/6DgUGk4B2O3qJRVZKaI3bftv3YkXRSBAYIgQDbZ5uoFrMYghEK TU19b4s3SwcCfmrID2WU6CSoLIljQf0iJkEuF0o8j4smL0qq+dbISIQUTa6mrYSf8XAFSm9F0HRV 3hr2x3o42VCFqEkNotioNrHohUIYfAPhEihqbGFNJ+1HQ/j8lg0A3jJQd+U8w/iponXFbCAPz8GV AUuBnguIiABRZfGtvJlve9QLtUZ8IU7BYoRFOAUZYAxdUoSwhYuNIAhgfC7CIc1qD8xT5ICBUQA8 uoAFCuIoh6L5KPAlkOTmCAZO09OENOpw339o/LSNgYgNkgjZGUCb98Ircbpeqsb2WsCQviXtETDY owXVKtERQpGhaLwAC5BzAAwsQCwVWrUSodAQR0QR3sIAIBe4EsFmPgACmkIjagpIwUSEAB1SImqC pugrAQdJQKxD2qkLUCthtQCtCIgAU2KxYG1gFIRIUobjZbGdDz9jCbao6DsSQPkKinbRUqLkuABY hYUufFS9hEuHY7iduNAT8Apx74M+boAVAMciYApBU2gIPIhEikEJFWIroe3VxFdjTdVP5QOqJIEL 0sQIEKUFCqoisB2UcXrxmEE4so79qI3xXaIOQPugW7RW2N7vAQBer7gXEQ2sVcBfnQREAD2H3Qv0 SwtCbAHG6iIAGDIFVqw2uqKax9wuHqcqZTg88AXUAuNINxDXyUAh8cVIKhRoClCiaebkiQKBS/Nv jYESMEfgvawqaEG0B7gEO0G3AkEBSjoPvwHT2ETVJsApQu+wEam0aiEQSASzYQN1XdQAQD2wGQvw oaiIgAfCGFWyJSG5vSSgFUsDcIEY8MxTphbainOQNEuTUBDAZIqoXCEe4UJrm+b2bGskQ7iA0w5I Fm1Ig0EVAoFYAsUWJRPw/ZCB5ECAgxBTMFQSCRUZDAiKcopqJcbhh4fIWD8tYFA91FPfpoohFKVR 34VEiBBgqQYHRpeQhd/kLuCayyCHD0TiAhvydkBSA6iGref0TcIYgBAyIED8Ii3iJOyGSk5ToaBh sXIH7jR9AKQATtjqdFH51HqewEEIisgBBDY9QtUtDcuhSqXrvr2TCgt/R8CewbNCiHwSCUCs1UYb wTV2FfQB0cKI22DNM0dAm/ohflQFIFgD6ieQKY9F+uRDKkViqYpNROgUKt1BIQgCwFI7z45DRET6 BdGZN9gA4O2UDAliiuP+fRQ9Ig/JFQQoco72bFiFkSlryOsRMPBVoWEWyJSNrBQgpQxEGAkHzgKJ 7xTn/+QVIeHkREADhVsOFbpZQFPPqn2xcydIAhkhQK/lbwFIkFEiSAARhFEgEghERIkgpBVOD5YY ciC+6Iks/SJybAIbZBW67gWCgAwqDE3f+6AIcCBzlPq0mHZso33ECSIEUR1DIgdRigAgGQEe0EPk 3+wJD4HQ7+CE+0RKQRDd4QAgL0RIagLcfr8zQHzD16BqncwAuAFo6AIeqZ95ANEXrbvZTP4gFBgB bneKH0uJSFIT60itIrLCYBKENIjcWAWAYqUm6d974tsAF0iK0qUCEUgLmKhSkAsIlvCAduprmaDR 7gC0hsIDdBNzdKvmREg+aP9QucYZTIPosHIaPxl9BFubuQpNgVdKHt6PLKML7lrDQUEZ8HcUDiyI KX0sqDsTlACEQ3SkgzHGgFJEMQD1wgCF7FDagtu0DU1iBJBlRYAhEIqRgABPv56awM7ANlPeYAnY 9O/bRcCVKLyEyqt7CAWBbmsA0IiuQaMaFqyAzCqwDSlAPLPCWU2ylkeLqbp70aCruJn3BXCOuR1a DUwArfgxqQFDeZQwjoOFJFNQMUbqmNbosBXZFeQDS+4m2iK1YJYU69RAa/PGV04DWwS4iUJ+ssjJ dGB2RBspt2IDQlhdSwc6KrdR+seXwRUkFIKJJCRRIiJAVIxPWh9gbgXEAygwUYwiBEil3Agm5HQY MSB7DwUIBGCIpr+TrXzvCit91PBEQoBDxDzkL7Hr5Gzc+4gQcoWbqLNIBaiycpFFbpEiQCuifMTg 7lCKwgDDk7VTX+6Nj7JcH6PlxwimoiQfwgvb5AwUGD1FUoT1RB68K+PIA9gFNjwIikgNovoeQHvZ RZYMFhLek+tlQauAu1ikXzDYLaj2QA65sXFKgh07oGA/tlVuWghSgQGCkNTwX9ApEt8QQ9rvrn4l 2IrbMCBFELIAZMZBWhCyrOR62sgK5oN1AUm6mgAkYCJgCAiUKkAWRERqCpAFgmXYoW08XXUsIG/y HmiaCoXTCjdpR1AF9Evwp/cqiCpFA6kQKAQWEAUdQo0gmYoY72TSmnUwEtdvzRVxv7sgrViNMiJQ qwrBR6S+VEifelEXc5F/gj//i7kinChIBWEo2QA --=-sABADkyuScs9CgvROcG/ Content-Type: application/x-bzip-compressed-tar; name="btrfs-unstable_sysrq-t.txt.tar.bz2" Content-Disposition: attachment; filename="btrfs-unstable_sysrq-t.txt.tar.bz2" Content-Transfer-Encoding: base64 QlpoOTFBWSZTWS+uKWsBSyh/hP/UAEBAC//1iQBcCv////AAAIAIYFle9EnzcWZete8N3j29cZnr A8B76+7zw8kkgQoEgU+96kPqE7WZqhbbRwHI+Wso76nz4n2lENeqwcJF03ZrAF2niZ6rtnHla0sS 1rY1u50HR7jPXDg9A6eAoPG4za57dpWewL25aGRVs66nG3GcdbAJXw4ZYvHPNPnPcaR3LuWzYkyz VRtAFWpa3s5GMee086uAa7V3bx7h1NgbvTfYZ99oUAH2rvbrjqszVeNAV60qQKVQpQSUUqI9DSlC SEACEQVPQU9RpB5QabQT1NGg0MRMMhAUUqGjQAABkaAAAEU/NIkmiYkIA0AAAAAAAk0kRGghEUPU NNAADQAGQAIkhAICGghpNTaSJp5JpoDTZT9ImgiSBAEVPSSj0TQB5QABoAAf8f4f186sicsCjdFU hBFT18UoKBI/r/eqipjCLvItskMwhJD+xNQkFkQYQpiKFAQUSCiJVJEpAWkIii1JAkkRQaBYIJeM iCH+J81Cmp9CzBDtdkAKhL0qmIoYEwCnbP0NWvOTbJDvsPzK8ohJF5ww1dd0NFuYon5wIdkkh+h1 t7erZIaIEEP4PTgEcpwFgRvuCM7mNHIKCnc1VNypemkRw3vsn8Lf5IjEdem5xuSEr9baKddAeUZH USAN+7J0qWVLpQeEJAbl+6UkS4om+uerJ7BvejFQ7pkDqRzbjhbrqzPdGKhpzLGLznFsPnx37/AD 3uCSkbKZZwpCUVtMwVlDPd79NadFvY85HRWRfueJViybK5h3nM7YYptnrPRigdtbFEqpY7a96NlY pvlt3rwegx0NwmNDNBtZmiAMw0grr5+GCqQFJFIKD47dvkBA9cb8grlFd1Mnt4vU0VsUxJGRkSQk WQSGsocpiVK5ozm/vldbYKOM1y7nfe/OJMSGrLTTlQZ6kjAxWGlGTorbsAsAV01eavd6bvzq27Yw IQoxNFERJCE2BmSSXZZHadguIq2BXpeURuCufes8e+lWwJmcFcKgdMVnbfGsQ5hO2EvLSgA8+IPd GcU/qjxm/dtZONxue+EJ0kL0VIHkKIp6HUq7jVEhSvTNqcnej1nmgkQaifBGuQYJNQvKrHdtcJzF GQJBYiM5kVPnwOcdXm0tpw9eWqZEIBIsgsAAhF7GcB7qwNNCNYfTWEwBYRkKhKSWikikFVSBSIIR rSCFbIgVBQoAwYRqCi0kakWWUKFJEGMZBkYCkSU2oH6bIFYQRndEoJITehz+pFRRzAREbJ/0PuBF SBGLe/fK+dfPzhv7z/ANa1eIta+tUtSNPWbSlWVryaALi2VYLvLLEG7YM7Shuna5urdbcG2ZXjk/ sONXdMoAEXlrqlUMKjVWIZ6uzDwzCUixMBXmxVVdiZeoppRmVSJxRle44n1vyKVx/3+y3lP908H7 2P65UbzP+uLX9If04HtHyrt53UCiBJOW3xT4za/18/VSp19ddJBPObSs26EUprWrI/JmQwzAAVFW IKwkPxSF6vST8r3/MeWB+TfrMiN/sklJA+JT/Kzc88r0Wb9hicozL0/1d9ftieVqjzNIvRy/um7e NJk67zM+Wu9285uo2zkZtmrsRJVttXiHNAAtdBKloOVZ/T1rAttvGQ7VVL6L7bPDTpq51V3tveNK 5f3qnYrSd35I4yYPLJZmYAyRYDJiIjHEljZIFaliExrVYUy2Fnr5LbAQGASoJtAoZJPNamWnOz26 6vvPV711jDznRzwm5MIDvvG91V55ozJMwPO7q/Q+Yb+uwE4LuIqUwJVk1qLdasGYaicDjrYqM24t m5isR4viEUAGGAYxDv2bw7GsC8nW/U4lLdXOmTj27ZiTx0/AptCsH56vPfnDW7wmvRzLBS0pO7fi zMzJWd32hxoKhyksDGbEkwhEYgoDukrOSkA/CQKpFFVVVVVVViA8kkkhCMh7JJJCEft+CQIhAWKC xRYsUUUfvcIEOLd26tkt9KIqqiKiIlWWBMqySzVUZWkkr9/9BQEHzEVLA/wnvrSkh/O0/G5+D/f4 +e1/LYPsqs+/6Za/HIrOu1p/2/cRUoRRUKVESDJJEEN6K85Ixi0IUFEJp9PqGX6HlCfFFyA4+vSR bGQn1KVS/D85F3EyTICjSWlFgVoTMYZJioLL2lrA2xtVXQYZ2q1vnW+gip/oIqfcUShFSAg5htlQ Z1TbaviqJtUPsyMGRfyDG6VsAmw4FszOwDmU0jkoG5ZeEWhHtaNLa29sK5dE43tSSXI6GcO1V+2u MbVcLK4VUdy+HXKuuVRAu3mUzmKGaNUMKghAsfUTFI3EdtiSrBAKtCj1UsJFfuAuBcIXRcWRoAmC yne/mAKekQXAADKMRJRhxr6+13sAzMycDFiJBARgEEk1pfIldh4q4qq7NJCbAlvVeXjy68vXqjpD x5ZnekjxAABVl35N5vMkybreAAK5Lv0Lm6mnEDrzdnluSSakktCcKk8JJLSXK7uWsu7mSSUAcSSa V3dzySSakk21HI8kkS58SpOytHRg3bKcrdXAIJJJSHdwQSSCSSfuyAOPCxF2LJJJJJIiIggkkkkk qsrl0dw7vLtyZdIFlTrZSunAkzndbzkBoAzrZvdzkU3UJNGAAANDAAAXO7KfOOpKAOkkmtAAeiIj UhwG9QACxajhDgMACSSSSSSSSSSSSSSSSSSSSSSSSSSST4REekk43u7tW7H1vtMvAAAAOl9H4883 ynyN+OC/DSZ4bzIW63S3sbe+OyGd8M1gAGgId4Zoh/vpN+oaJNCIiZEkloV35EREyJJPBEQZEkmh EQZkkm4iIMiSSCCSYwzK4XNE6d5K4vdRcdu+tXzAAOAIBAIOVvJd2BABHULjc5TLwoOEagwACwIB AIAAQAKCygCgOjnOak4b34AdNJbsGkurwssKINsINJDBdbk3VEU3lEUkukoW7LuFuy8u7mLdNeZb VFyeF3JLQq6zx5O7aUjkaUjk687u738Hd31idu5mT4WSTJAA34AAEACQABuw4G3RnZsorTLSRMWA IiIiL1RJMxERqIiPREQYiIiIiNUHA07OzL+EbdW7oGR2eLuKm33UzMn8RERZYAXDhg4e7JJIUgkj 6qQlwIZmp1jn172ZgAAIdLsU63ncy8AsAKDgCDu7u5jinl1JJJJJJJJJJJJJJJJJJJkzMyZJJIIJ JJJJJJJJJJJJJJJJJJNEkkySSTtmHncJ9fNbdzl3d9LmbkkkklOvkMBVUSSSSSQSST2IiDMkk+iI iZkkkEkksq3DRFxEQumt0d6d3hX5z0UoLB0dZh/YgR1WXDqFdsb3ZN5DADquFh6iSSSTyXmZmZmZ mZykk7oihcknh2su/EzMMzDMwzMKkgkq4TSQkmVs667uJPd61dQSUJJoAQqJXcOiOqqqrDH0tAHI ySToBwNOSxJMEsSTEgCJkOBLzChhERERQkkmgwAqJX8EmKoOBTksSTDOzGtbRUVAmGZA/xWU5y9j LksSTG7JIVmZmZmzwtz29zm87jfZmZJJJJEREEEkn0REYb8Q0RmWAGYzmZmSCSSWzDQ8Mq/inzDK Wd4dlROojMlIQ8ZJO6IpxN7u47jhBxwg44S7u5JJBVXZEuC6kQ1K7iIiIOmAHItUTZJhGdX1CFXW 27uHlmZmnacefsp1avCoyLbu8MyW780NVhpBA1PT7lIuw+Hft3PK1bU9VpFUdJvaA6sQJPNI0sQl UJKoIJmKqiNpIQzI1AsG2khCkhC2oFhBCFxVURooglki1AfpZRpAFiwYjCExkshESQZSCQpBIVrJ CylJP8smEkmMkEREgoKRZAGmmoKsBWDEhUFkShJEAIisAVgiEAIiREoK/PRD3BWAP9hIA/n7oIfh S1wD+P4fypfqAORSimqKxMIB+xWCAarKzcWltwRxgVb6EBhlCXL2GC8Z5E/gc5NhKc71bhQEi4fy Wx+hg5b7XNFSxxzXUzMaJYB0siY8wnQZ7MyjhzcLhfCIxFeIXQ1qb30bG+Lij0CtNLbajdEdImqD LC4C2/mmyGDN85LqdE2T+MG8SQ63zOmIYmoivF7j1gTYtiHE7EpMW2imFgzOSokFEtcFc8ZGE0rS ZqjnOr/efGL6KwP9uGjnh+okFMgvAmdlp3Oho52W2mNi1bKQADAkr//Pdvf8aL1X5vJqPv4AkkiK xMqjAI/ryvS/3Z3PgmThEftH6BWgPnhnUbd+Yw/e08rAAN6ZInQACqwWlP4YmmD9f/NA2mfIPndU YKJ+fyIv49p/d9fb4BHJEcPSxT+25R/HHH7G4nXO5mbIrEV5N6y+csHPgyVOFS7Rjy5PC0EhcDB2 KtWwPY2ntkdTkzpE0ib8DaBcTyCsVIorYQClQxGXtmSfZMTJ7bi5AGcyCiBBfaeTD1c86O95ZEdC eiRdkRh0HsltjfXhAoLGaoMYAHAQXBJTMVJ6y50zzmCWkV6A7HelNtHEeE3ktdfIBgTyYfaNRAbJ EwB0Pl0R4TAil9hRNFFeLuYRWCL49Jzb3kMe76MXNY6PSkefPMxZ1v2m4wUiPPpVm1CmEspDhwb3 JZxi6VQwABLeL1mgcQzASaGFMvQwJuZKnA8wVtW7g3cE5iWe07IH7aiIi/CG0UEVkAUPsC3dFQET jFGJ6MkiAQudMFHeIGOuWpSfkIQTviJFO8SKLS4K6XyKpoorLiq310UOAYRUiAH8hL/3/t3e/XOd fxrTNGj3oEj/LQzOkGNOgEV5ic/O+qPm1JTb7uDXcdA0moWIgJA7A6YaEUCd9wexhFGQYx/I2CV9 UB/7D2QAWAGSJ472qgvK37Jmkj71hM9b9ez8eIGJPCSdavdpUuDCqyOGYGJFnKoQVQXEAqSi6se1 rMZykkxhDjdkNsgdY4KZUaNFUhs3rIsleEnDiaYGMOmaST8kCbSQ5agskrt4fabB0zbzbpCvd9JU FmPZN8Ghzei8sNpOzJrbbdcN1NIGk1lKmn6dM3aT36rUzLcn8uIkgAGYIvMUQ4imfFoA6qsfhQoF ggqJswB7Ai9E2ERF5SQhJO+ru3Bcy5aUtccxczMWNxtqtlymZVxcLcuOXFzHBG2MxzK0YXHMopmR FSisxy40bLFyYUNiaGNt00bKkk0qBJodAFDGwKpgNsSAG22zZg6G0u1MlJRoezwrg5jlZZWraIuK ZiVctzDlcu8vGGGS5LDq3nN1bRBx8v+ankwZB3gBzCd+t60TYhhci9J5Zc1hjW5nnWtV1aS2FSgR ttxuh04gdRKgFUcGqdRgADQ06TSqgAAGxsQJMbbaeY4LctKyllqY55z66zemcCcVy5mJRMSAKdAM YnWUwS2hsykmAAAAU2MpsAbdFCKLtGt7zVWAFDbEUNU0wAE0PZtjVNUxiKAB7NobNA25SkEmTwxf cumEkkhLy1X3aV/NPeuLVpDXwlrlvSI1nL+7bVfs87b5aLq2aIiodavlZxqrbq1tXxSz3a85AYZC RZvbdMKTJBrjrRJN63sopZIiHwirURAmVlOSkzCMOOldmYKhbVAgTesjDbRzCdQDYNSiZ1riLWpl WG/dyErS02OVXtvwcgUAZEUJEXeAJaJ49gD+kFTcggKekFRkACAKkQRbm4B9n6biGUFTuiD8yuAC 2J+XTEyW1iakuS3wzsb3DBd1g3RPfJJKRPzkx+EdzcK64xcKuapo/DQBVyIKhiAjYQGNlgq2Cmiy ollpGBBSQEyi0w3ZSKwVN7UkTFACdzNkkiIqCHoySSZiIv3LRyEAD7+bR33VbvESWQMTe9AcwXk4 MIYvghUyiXkDSgqIqqiIoqqoKsVBVVUUVVJglRQmAMCqy2AurjhqSJMJdIUmfBvQ3Os6j0UewPCo CsaMycCdWb1Cj3Z5Hs4fUvg8CxRVFFBBUVFRERRgoDBBBggsFYiCo8rPAmghxCzK8ircJCzjCZNc L3jbZqKbGwK1EVmPOnjvtPSBU0hyQ3DpaSoQEgETmLewUjdCKhdU3MHMtndoOdYthZIRe/OLhlMI i28vY6O4SgCispYDl1RYIZERAdSIbRSqo9gxxsa8GfMpEcG50oJwvMHuatWpfvXmypDC2h2HZL9D yRTBcoRcNuTJhREowGptU3eVuY2LxkdlXHB5dezNqbNFiy0Rd4A3SJNRJXeoFRHKBveLsVVMF0Kd CkBQlE6JgNMuapR9xQBY8cWdqKd7TCZtePYQKP+aiq9xFTuKgUEUWQioyCAkVGEUWCRhJBgJGQSC yESIyCokQVkYMSBCAsUGECRRiEIwVYEJBWAMIKRUYkUVIMEBjIQgsGKCwGRgiJBCKyBIhCCogwFk hFJBgyJIRAASBBgIDBGEWLBkQBgsgEZIhJFFRghJEUWQAGSMiDIIIIQYwQEiyQAiwGRIqJEViokQ QgKwJEUSRIsWEQVkYxIqMYwEFESREBRIxFgIAMEgMIwIkGJCIADIISRFAIpFEijIKAAkRkkZILAQ QGMQhFWCrAVhEgkIIsGArBDAMmVph6P5N687QAYY9pHQ4G6lHhXOFUiMERjDvv9Pa/Pnztrx35y+ MPEtaY+h3JWXbTbrjc16dsUnmBOcJ0+BZADWKpcdY8QAUtpDlmMm1iVIFVnPk+8kPR5fRdPM/cae kyiOgoFejz+NoCPEkAU1/HvvyOOwZ6R9+B55LIsORdBU7QzIkDivaIosjIIq2YINQEQaKoGRH8mE wiRkQMIkLSqQYxQh1lgWEiwIrURGQACQR1iiLUWoOk+UXgQjaHg0ywaEXf2g3LkIcwaJBjsMChDZ cTVdX1oeQvGSUhSVR6EMgjBA2DkOFpLxOOeNvmVAS3vAJeAo7Bj0iI36uFioiOFVRg5pAbwkOiLQ c660jhIRwlyBb9Ui1BCByayC+iqojunJb82OPweP3TpxJbX6D0qLZEhzz+BIE/bKfNGSwog/r22s +CL6fti2t6MdFhLYL2U8W4ERKYLUFuFqRNRCxYgB9i2DxDpxaC9kZJoiMK7BgrsaPn5XOIoCXi84 IrMCxKScBJCW4UdhIBGI2izrq4kLZfR+sZO9w2FHooNnBiwW+qnoqqI7Z07IQvTIQrB4HQ/dc69U 9+oEY13dKHaxgOaRBpHbm8MwkYCYySLsm1R/W8ViIiIi1Kd0JkgRsnues3qNTnWu1qZjfvhn2tDP Obv0QJr2dZMsh4rx1XL+jWuVVbOaqr5vXGtBPpfvM6fqInxMYwi4vPDt3d3d+E6JMlAgjpPPcZu0 anGc8tmtZa67pXrOezaCSESIKipqPz+w3qwZ513vMeOKXryZkSEUA610BKd8mxipCXlnCRvkEZub 28Zkv1mGhSwMB7EGKBaLkA3YAaCxoosRhvgloYDYDWgV8AI5J4IePoE4tVdHcmGI4doZir1h3uXh 6R0VvDAK9adt995Gb1UkhlsIBkmIKqvbBObo0Me5LVU0HrvrQydedXvJIcGzAk8KeRhBtgMlU762 tjeGyJzEvMy8yiOiGCYc8YAmC2OylIl4d1VVZDR3Eda51pVNnx0b7Bs4BzyZiroDAE9uXhKLNrSS eYbomJ1r9uTye3MbByEQOIvcTuJYgXYBnNLxEtvS8iJdpMOzXgMsIsbYbrlN6osaSQijBpiNWrap YwxhlwJnJNuKc6Rdu11SEW6VbCAX4/Gb5zfM7fTVpN9S9X6p9JsRXIJmXXePGd3xSiQt69wmQLvF Ay6X9g59KAHpBA9cHHugeIhxDfCpwweRIQNqFp101VXuiRQHrmwsck677EhDN53xJH2llg5qMv7Y s5mdTGsB4OxIcolSE2QLWOxM3p4NnM3BWjLPwSwXP2p7NThRU2+31z1kdhwosL0BUVBIIjtCe/N2 kiTXJgoZBF3iNaiA7zlPTTV1J3hEA81BVoMDj00LA2u7TWAdCN+Der4FgCWCGyWoyHBwCsbmdu5w KqiNIrkcAHNTYe+evCaZGNzcwUd6VHxKX1NIEuB4WLQiOSEka+DCcVAGUlIByMBGzu+00hsiOLW9 5zQON9hAZ3ko8ybhwJpiCvDvqiwOML85S4CuAIiKcon2tENwdhVocl779XnaxQD0Uzj4IseQJNgE igAEhgz5Vwm85+gfZoARfAg6+EzoHXoXYi3S4AE5g7kB+tE/PsuyqyiPD8AhunbZdfRrWhya5grw AfN+pO4GsTzEV5JQX3xZJMqPJLQ5khcOrh7A3yUHWbCqfazsI+GtZ6iyRUpYsqde8We6RUCGcHDI yQigEIlo2RVpQXwuP77WL3CtC0nNKb3+GAwiPfgAwXCQMyBooJeFBCBy1Wk6dfNgouguyEKlQMVO 8vWviB5Ehzpvx81xj4t3iKM3125+FT1XmLzPfXng1Z+gbF/x0SkabrdWu0JHnYi1g4DKu+rMx8Ux joA36OFQOA3BFJxDaMSqTQK9Cpi7t2beUnff8NFnKoWQhgYuXgezJFZsXpAYJKC92xbZJGtJ4dNL O5kAWuMxKBkBScxvZQLUkgIxNEdUWSpBSGoQiQqQWSwCRVRr3w3BKLm7LClBum4o6yekW9wAL8dD c3vzZo5JaSFASp4+t/eBThYDMwJAkrU7fNTKUOrOYk9kG6GHUoUwQk7X8xZEcPWzT5QVeHnbarNj Jce/KPgdAQ0vXGIxk8JdkZXalN+SFuet/sCO4Vtjg8Mbo4+9rfa8jzNc1ZFbFPqRXMVUd8hUm/iX PAVmQ6gyLcifD0tYfFpfpvcVVEcHKfDHvv3ybLwk340G/sHTG3TTXnfnQ5pFS5FBmCgG9gCjqRE+ jldjFAFxEm2QSXVDaaLk7sAWPXB4LsEQJz5kjMBvbGORL8ms90KftvnzhDckQFIAG2pJDo+irgUT hyiJjGusmNzuvs/HOn9VKXJmZWZmZkyqRAmZMq23rAzuhJOzDTjbXuaQYz3R3Zot05KUa3W/RInJ +6EEPSkpSdn99pI0ytljx9GbSvbztNJCFf7xKM1pPbRP1z/j91l5m/vlPbVrqGTMHGDjC/OZb7tz bKTO1iLiwqqI/DO4HYKw5BFTcz1uZl0yichkCEE3KpaaTve8FFHjZvACoF1CoKRij47k3afjEtvk 0CbHCFgCgF1B7OPyL47fuCue/R6OayeSn6lazAefPmbbcJ0njX9y8r8nMXtDwc2u5+K41z5vraXt 1luaOo+TCa1O9+6F33Pq3Np3Kquk0785tupPA91UjpjgPH3fb17qcjnvLC9V9b2DVI8d8ne2rNm0 jie73Wrz42zedVhw7IE0kAhJt9PCEYA1fP0y4xmxVQ2vC3NTian55CmFcVl3klOcpxKUSnisgBcJ aNWwOWt5riJJ5orcuAIFpkQSBJYhZPEbCZ1XMsgkuMLhXyfOx35EED3xfHO3AvvRWzVjw6zvbu7u /OQFb4/BlkAGGBHwJJJCFmXXl63pfLPCYA2sIvcxoCtw1pFcjguQnYnll39gatuXnvCSRgYaOX4E bXQy7DSuq29yVsP2mmjmOjpnyBQEZxkSSqBwt4B4hCwZrPJFBZKd0kg7O6HZPQaN3x1MoSwlkTHe kFu2O2AIq6Qincp9PEQ5KIIkM8HNFxALWu9OQ5fOMxl2lKQO8olZCFJ5li09HblDM1RImaqNBTTm qykTR3oB+htuDjr5og2COWfRTwidfbF4rnJlS79JVkRnUqUZ3j71Wsteu75ib92t9ztK0/mM05xt X+K9z8v03sm4U3mlXa75Mz8GEPlMIEt2N1K9fDZamuIDp6Hxvd+x3K61c0fBDlhym1bnJmtW225D KuUpbVXsdGtLoS08dGtaLVLvnNTOd46GsS3gcy1K2lKIiKsVRGI9ZzvOjVx05czW+FYu2nFlEVVv syKa69avG0eKVWtViiJWiaPxuKxbdGW221tS/UAvXvrdOrnvtnKttnRhmVraXbVRVFIYKiir6cTa HhbcirDNar8udpSdTQYbWx5VIiHgmdMmn3wEkgyzuQ8mCKKvRp8vp1sVEebVURij1z4vgTMttzc+ Ib69nsJJ3opq1Rc4opq1RfobsVERFVXT238wD51xyrLc1rWlpbrs4YyVD6nviGw5nR7JCw+OnF8/ AuONwynZDXemHbnjA3LWedZQABIprzwxn3VX73czO+97280/Pve6fvH3UMva7e+THDmk0zc9GlzF cTJ9fNvKbW4i8yeIiaXPeMuwACMtnvDGedVed3Mzvve9vNPz3e6Ft9DRDM8aUdXDCsUA4o6AS3II 5yHjSLnanREPic/D2uu5MZxgLEUvPhUSww54DdEafWc57EqSJu2XrrbPpXtyr3b+J47zaavWjcAu 58zdbZiI2HqrVWqtcEfpxbeZ2IZ4iGeIA8jKkpPsIyJaMqrgalk+6TBXfwSwhohgLQVJeWmWgNEA KnmWMBUwjp5PmFAZix3adEatDONCF3LKV50gmslSebM0RQtvRW6ZrOc58wgr0F7EvVVf3lUwWlmD WnXcxIcQJ3r8et18veUnz8XtHw1D21unwyyMHzSrA3eZgUYF9kXEZhBBCDEGKAwKFS1CxYFQsXiB kLJJFEe6onbg5ODUpQE6aAd8gVvaLtz2QCxqenuAUY/Dclkny+YisTdpSlOZIiZWSR3nuykihAVt JSINfqUJ2grA2IoJaC5wOtqVJB0YhXodS+/PecDoPaCbwNhILtFR9vQKkiJIm9r298xaqtMYoqsX xjOdx1pXqRMUDaLUaIoLV7bmyolsOIJ6ESoMiRFYBygFQWQHNUIEiLkDXwxdKHz9rhdMjQY20Mqy meAN+5qnb+EO7wxYBJCVdehqc9eCAUuCEuWx/3pHsr6PLgko1TncZ7ZREREPDMErKJOORYgMmUCM EXYwmMJhJGkfCAIKdlTXBz0EuZ21xi56nMPwrSzHwhJKFbwA3gIE4+ltixQ1Cr0qkiA8822/IKfM 4MdyKh82C9X72khUqQUzY+7Vo7pEt4pC5wBtf4qc2VX26onxdjqt+jnCCEiEigHHJRaKE3pTfa9h UDUqCEsgrSiNQDoIFb8JDVVG/v33q/fhVYwXCBAwQCHxOd1jWIhFwmEwr7QYeYYpvOdHUUm9Zt2O s1rWz4O87BVF6lzCDLz1qeS73HKce0iOMGdIr1s3sCvvgZKVE5ELqidQJCGSwwk3TV00Ul94p1ih iwzfcvu3VjXuLTfrETis++3YGpDuyZlWKUEcNDpcJ1S8s+vpYK428/AOx2SX4j+GlqKIQharWS8A xDGVmSbPn1mqfgXJJCfSAUQD5YVAcUFBxABye0ABh++66L4R5T2j5OPKBeo/eKd4+aibHO2K+SVT aSqtL2AUwMV16KJzYLHPnW5W+AVydb0ItRCQBNETeJQEG0Xf4d39A7Wc65euO+rYm927ayo9CZ3q oQpcvnyQjCTIYma8rh3dM8MV0+wejBzvHBIWqmd47KHmeZc3eVoeIh4iwC/QDHdcEhFhCfQFaKNr iXA+5zYFYcwAoFZUQ8guouhKUptBPYB7A3DjNFl+QOAVg7x0fesbhv95QTYMMBXuG7dOPMSEQ3XI M22ZzvMlHhaWnepUrSpoCCqojxS9c/T4Z7wKaOD18RMZdoRZdDCKI4bBJZmt8PMZlmxGp4jLdd+T snrvIeVQwtvhOzz3wRpnjxxwtLRtsiMB9jaG8cIbGYLYPC5yWCEDCIxogb7YJRKs8MJkJ0cgrl9u 5T1UhWwqIlt9vZhFiIEBQAJKu4hyW+SFVVRUmBa3BGb+HYKwOytvdW1rzPy7e8mbuchCnOlCyKPZ mISrnb1SJVrruQF7XrVTQhSsfUzzv6mb84BWSJ1sAtcnBCbGHAVQMCSeAvCIBJWvOxXgil/FdCFW v5fL0pPfvhPb/NK1+dy99Z+IzlmbuPVBIk2eNX405KYguhAYcWGkHeG3dDXi/LPM+BCKpI2o0Qg3 rzvWw6YS10GEEtRRIVIKHhlGAa1RYCJygHQzbJFjEmoNREi2bY82/OjoCKZTS/uGROIqdWplrKms z7y9xmZkmZhnd2Gy/cBcyd0ijaz1f5Dcvr1PxK98WOfLlpTcCOKhq3Z6znWeEpvtJsAo+uM3N0vL 4jovZtVDaJ1y9ZrcS27vL32+UmfnfAoIzmxYbfOefuc0yMozwRJRe0qqqLRhCZNxCHi0gP707mur Bu/S+jrmpbI/5oXud7EJJr4KJwaHIQ447/pUvzzZANEIwiky7tyLEPb9ufux7ddeeu2xOSYX131M gz2aUVgIdzv8+wE3CbA2m/rxutbS9w+yQ6UOxZKMEnP0ZDCTGSxUBMSO3PbBIDGMDOzM8ltIygpA uGRzr0pFzBZZYyVlsfYzpmu1BpIWJIfXnuiucLu29sWIzmgqNUax9sYDiNxtABqgZGjTRAATyZYZ wNzYcDTAYEbBJQR03IwiN2vEZyzMzzm3rIbUhkgEfpk99CYBNfhUJJZOsGQIEa9VgElbDk+xF6Bo aijJBI76SROd4AUR3dyoM3pkztdF5pVo7LVugwIqmpE8XjGMzpjjNNnbQaBhIKTfKCXsvir+zs7V 7N2G6wgXUcGgTO4q9OBlSYl2YnkuHVpk0wjsbzDo73VIYA0cVJSQEQk0P945nrvrmOUpakq/cWp9 4lW020/bYfLeomeYqcFrUicFoYr5zYPucmK5hgVfozJyKawazCbXEP1BV/juHgYfP1wqR2z2flAU Ma0MNhHi86oIMiuF5Obj6APIDOuksdo2gH7PXKWOKX678Zy9EFrSoWsjG3bad836tp9GIGkNjbbG 0m2m0AAA0mwABsYgQhNoYCBiQAkAOmwAAAAG2ANsSAABtgFAAJodMFSbENsbaGkUDopNum3VFXDL jhhloqmUwaYW2lwtLhlpccrbVsKwzJUzC5ltVoLcHMFFxylcuYXEqy4ji4jW0pTKmZcwTKWlGmOX PDctW23WY4+wNTyHHLT1RS+WAAoCUaNrreLzA2Y8AdB1Y2jBB5Q+KndwuQDPGjKpxgNo/CE480DR owBED4h8ePWrW+7jmV4LrLH0AcdjQ2m1AAAaTYAA2ObR1KlKhCbQwEDEgBIAcQQAAAAG2ANsSuSQ JINwCgAIxR09Yjbjltwto4URzbrVNHv3fCqc9oCwsJhNJjTaTC8ymcBu3xexnJP2M+ix84q98Rzm ubqrZt9iLzrarLcWc8arReqAm991eudldNukmXlJKX9PERE3p+rEbsnMJ3sWPeVfeI5zXN1W9V17 cub19ISiEkJALoIb4d173OlTRq9YNNe+imAPESrZsZtn15QsGD3okMln2Xk44zMzPc+rUwYDGjA5 6INCSmUEVBJaDsEXKHk5YrWU96fmAElOdb3ra721e93c3bZ1DgzdG5u1hUZW3iERPiDmMIqdwhXC Z2nXMUBcax5p0iUhnBCnAyaJhPkuNuzw4iMmonZ0EkgEdReZMh7Na87tLVqyGmIVPZARPKIogD1x /Kqqo8/MVVEaveqquYqqIxP3EUVDruBy77HUp6PeMNYtlE0/BRq4358XgyLsiiZAcLafpBC1fjWf nzvJsb96LIjx+IjpEfFOz32AkZfRyIXM2ak1x4EL30YddRECiWFw0qX1cEJ+Z51eHVQkEIyjV5Qm n1bJWLnRZKrgRblXanWEbJdCQi+gLaUt8HiTIrkV9bSstgCnu3QkPjGsFYEksSyachCOVS25ZEa0 Njm+/Ymj2w9+gPub7vpyS6J8uPLJGOm0wkllGSKPtzc0Ny8l1GUJLGSMuhC2AYQitXUUJGjOyZM0 xMCpVCwK0J1ucGrdKiX55FE97TwCwEECkbhsvCbinSnJk8BXekPUdEE0B6YggPEBXdPUejsFdwV6 BWgDVE8hQEH/7BFT5/8ICJ/xAUQCfp/m3/j+/f9f9vz5A1ul21f927u/4CqoggiOWW+d23UJy7Zq lAA57VZ24YZlOeAz9IZmVkVBE/cRRUP+4lIq/b/lBPz7/3gfzmiv+aUH87Sen71IiKxIP+x/H+xH +IgFv+8uXSP6LnCnHOnY1g1/vOrNxEeEniZDxLtu5P5UWyf8r+vYgOHSI6MicFcH7WQW4J/N50wJ KCv84uQlVdqqDxZOTrFgwelNGxdVwSOzEBOA2OpqlwL7pcEvtlSonceNBTKI+1w2Mm4K2RWbIK// bArYFY20iA/5Tr5t0HBJN3Vrj7QAcO/RqFmIeUEaNWulPAaZSSRQD/KR1ime3a4m2GkxoBlv/nwI cxf5biI9zhEfbLn2nIzJUQzir8KCPjeF+YnQCjMCybIxILcDcVdixITwdW+ZxyknlQzUK8LsWywP NEeaELWB0ip7GKln2VT+Y79XLrSBj8kgQvOfxiUHTuHyevA7R1oAl/w7IQuGFNCE3GF8P1TZuKmc 9oKlSrPRUTwegpqAmwnQ2dfW5whopQqwWxEQHbQVbY5+rSp0eAIZAsCv7HinIhcsXBW6GkRyiMRH jKCGAL/F7Av48JgfSvT3gwRwbb9B+LftofzO3/QToCv+N96Mf9VsAUif2oy6skRJ/31oev1uflO1 NYv+bTmUMf2pI8RnBUzDdW6AA/kIhEaLIs4HP1lCV7clM2H9Iyqd5A7U+2yXS4K535l1SbuuttDi /QUbl5vMkxKlicXaBJGCqzkp6Ekp6DmE9bc55kkhDnoegO++toomObW6z36Akgh10RlJu7UkHFYK kBlpGy0I2waLki4JAwwlPa9+F02bbZzQiUQDHEJIlnrPyycQQzLdvcT0PjQBjEQYvIak4xe/Xxxt zyB8tpTraBsgh5AsvSY+N8NjPK6yiPCI1x1A0U5KnM+3fg1MVpXPimIIQOnV+RoqEJbrAWR10EB8 b/H+NTr59u57Evr9hv2yMCVOY8eqh9X8nQLgr4nPpnSgKczsHjrTbKrM+XJEQ0T2p7r+VyT0bX2f Y8Q+OETTlzzxm+8GsZVXU98jcroXXYNA9kA6p8fg+oqqI27nJf1EFA+n5/UOTNSED+gipgFE3dRI EPv3IKqiOd26fbRF1zDNUwubQ0fr+/VTm79cvt3ySncOOVhFuJAIETInsmkqSBtr5c0zM7KMEQ63 UEeSZ/r+pmH2SfWxbqt65Ce96veQxiU5mWM6c38fN/ptVq+32vv6iqoj8hUkP5sZHf7HTy/Nbgjg /qPYjy4Z2ySYDzpzHqvp/emcM1wIRSh2O5OfGRA789bBh2pGNqRAuQIRFNzjUzZ6TKKQPnroibr2 asHC508gq4m1pFoFk7ZfGnAXIBvnRXXQOgFPasbwxM/L9+ibSz3FsRcaDth5zjMohXBH3wfssIuh 273rz0ugr50fWAAdIce9e4t4sg87hfBQXEYsi9/KBrQ0msdKiWWBREPoenpw3weRy4Asd26uTNEk CDPthff2v95OqNjAaQNoGmanmsxyOy+4ePAHaBrsG0c4vlRJCLmvZP5f4lF6P3VLpC0H0q40vqUq L3YAskULHcP1q+jlr4lVyHsR339W+cxkn7KKnVRUzHxsgb3gSJclZGILFwVP2leQTVGJlZx+mdoH fvhZCFaKAzT/ToXfqZnXTONaXjKJTJt1kbZGbwN2hPWbPWta4nStwYPr5P9hI+FaE9/B3ufmC/5x HLyDFDy9vYXBz9wAL3PELrlbyuD4lvLKe/jetvbGNsxR3IuzlrguYYxY5dC+C4EzaK4+7XrG9q7S uyyYrCRRMjmnrYZ5Q1iAFQHREAsMlFpOpWu30f1t0hv4eaTGVgwl5NLPoTMebvSkqWb8vfZ7k3tp 8mNsCWX308T9JQqiZ1+J2yXQnLzPHCPGlpooxJiEuF1edcOs3TIBsQ+sza5fustwYnOstLu98RkA SJp5KF3yYhNGtvDhGsslWtZs4qRaZ1VWW0vK98cZ1ozaiiFGxuP9kAHwRBYwACREZAEIqRRkQQIJ FAhCCQEIQjBGKkQGIkEAZEUGMkWAIhEQAVVIwSEgICQABYKQWCyCiyQFFkIAIKwEAIJFIMA5iVEk ADzrNlbpFgpBSBAiwQkFGMFjFASKkUSCIq7cBOIwknFvA48BXd+Klw3H5qlBDDkFdzf78RHzk/xb fhne4vVcUnigzMMuARftehplwR1oOAE5owwMmPMge94xJIpt1tg4mCYHppxhRJIoL5HtNvPPnFFX Jnc+3oSGkvIBnPIqr89IKGNs6gfu6GzQcpUwBw8EApAkvk+fplAoAgqFEPoD7goPbz8nj199HxGC OXqPEUB+sPQBYLc3NvLTjynKHiArEZeOG/oni9ZelrwvCEqVPTv3AOVqLaH8NM9hSq1hJRKhjX82 4e/pl7cYLHq+yDl3x6bVpF0EVITP8Vi4Hp7ijIKee7C4ddqBOO6FL6x85+U9zmPmIj8CAEU8Fgei oIeTPURGBUbLCfOToDNCmgFgQACFUlTTDYspP0CsIJqB7ACZEgAGoKKeoIGBU0RRyU7iHb38Xtc1 x7CpU9sif6fSiLn7/1xHrv+H2ff6fp/DJjFee/09fr5b+gfe/1AAx6eun7crV97+V80/wII+UAfx EEfeZIeg04iqf3AgF4oaNUopqCmJpjoDPf0fM/H5/pE7z+xKhRX5PB9/dqjfbuM54I3ImhCbc+F6 dEZ44nM/SaNfov3jc/hh3UWeIYrWPM7KSIms8rBzLM03dmKE23eaEJl5LNOF4KVuCJx3FFetmZnr nlprr2YV77TzuVJCE20sqJbuKZ6AjukIBpjjSpAhCpoZiqyWXdWZWmAMwBWUkuuNh7RERGsylKc4 ghtp7PRJrO5m7bPYgGntVa61sLVrpsnaS2WMhxkO+RdxuI1ujdphnDS7u771EQAjuQ1pe8iTWdzN 22dpCQgKDHxMIb5mVZWVlZUOCiJ4Fg6b+Ij1uHJ8a1fjV8irwdIwTZ/QZ/p8lkq6onwU2u9lrID2 5WsiP2Eo1FX3UzXA9eXUDuVyql6iIT6eZun3+E+BbQvqE5CexBJzj8WO2VsXh1CQkzscuZ1ceoK5 VycCUaSRkLC3BPj31wXMJazKDofKp9Xzy/kpPEujT0pSkmeUSsOVKiH/LEpVSH/EIVtZ8eQFsy11 KAvTB6e9l+96+Tf9rPkqc/BHPzDz69nsJ5cGHn8sohw+W+2b3hm+zf3is+CMYiWLWuSvSuH0j2WT WSPwl6Dr6QhQhKf0UOBAbEYILERg/GqSu840CBms8kR6qJ5yl1LdXnqy2RIJVcxnH2n6eZ6bfTbK pr6knF+BV/h/HGOsx4g2iNou16CQB2IYg5Ziqoj4AHqd5+qgJ+lFAMsY4+wK7mNER/HyhYBciHi4 ynF0sMGaFkRyvcMBmzCI2U3LqmiowxQAHqqPyX8dVR8iArA+zwJ6j8ky+J4v5m1gyZhhm/Dai0Zs T/cYmA3Bhci17/uik5W1+Wy9JY7eEVFpkRFlOJ5xvituK45VCogLLMiuZZUY73veuHnQ8c861vi7 5pTLnVCSdeFVVXmbxj3o+SiYxhttO9sqq7Ku3yOTLF4LuRKvLxLuZAVVVggeZhyYkpUousJhHTeA nMJraLLdTn5lEUrvDvXeMszMNwK6rpU7lQZmZp1brXcOzIaXgavn008JgZekaJoM1dpJO6s1qGZJ KNKwJZFhIdMLbJ8UUzGbt3cM1qGl5kuS7sJiBBkkBZiYl0VXACCWZl2nUETvGAuMWmdqqKplLXJM K7JEO0TSqqounxgcn7ViHMKKtF6JxgqzSjMq1CXlG0qUJSYwihUZOTyLdx2NM8DPQNl41LLyltI7 oZ8NPIkXdk6xQL7WG4XUb447uO7lDeQGVRDB3MqISRLIj6rRpEeLAj7tmbm9gDW5sCv5+6+/oetW tFPUw+0iC2a+iUZoCfAGYp0qXIQZdj+oDCkv//zQzX9SnMCcBGxJLIFlEPn5BX3h38u1e/P599H1 IufkxT6+3moqsoo7gf0FVRHP789J7r9uR9vk8uskhPILB5lb7+H4v4kkhJuBkzbNCxtzyl5kvmIj 3H/g5PmBocveZjeHUONg5adJIyWfXOAREf5pe3xRL0HwkbwH7YBAdr4+KmklIR9+jzWwZlMuz3o+ g00Cj4fSjI1MQoDmjANS0k7B/ATBsKIqMB0DyHg7iKWNS55JFVDm5gX+uQKGWgiO3Sax/ZaX9C2q I/Q1gK4EBugtY1BE1frmDwJz46GOglj4oSlROTNhPug/kzEB24wbOQAVoAzGh+FTApRSpQpcDL80 V1DR2hkKIqPZUsavTLJOu4/r3AsKj0+FRqKmB7BvdRU7LNdJCQZAIqTqndBe1k1EE7AaL5ncVbWc gNxLpiELAY5+Qv0PoHwFKeu6p8fm2VY+2MENNKvP0lz7pnkBopWLR+9V+chsCuWlGHYqo3hr+muX 457c+S/t9c8nH8kh5PHm7zwbV3vDaGhVWPB8kItIQs9AkugFnbCyZQTC67seWv3IwO8CxX8C80/u PrwJz9+fhO76+/3DW+/GIqrnxOOi1rd2fnYQnPG9LUkImNQt1gE5YUBn1StAxrq2uzV5fGjOixaW xVVrWLMwpDlFZsEAeS805DxMiJBVWiZMuIRNplPOiqq7Zk2VVQmAFcO4ScIYJbkjqXJNyvItabJa 5mx4Nz7Azz138edkS3UzXnvU4NmjiodPtRVX6JvSQx06gXYbAC6jOdYbETm191Z6RWcg3QTEBQWm AqHJFEDa2flJkmudAqPPvR91hVExEVA3sFNinUETdOSiRwgIkYqi34e+J1W216l97Wy1vbISXJUS qqgdAjyqqAJRmKBWmWLJjKLiMY5vqXDZDop+gr0wT3FAc99k+77lVRUlDSh1qvv2U+fITdWzNkMj ItuyZtJUeKJjwZkL7bRDbzySqqqro2CcfAVsnvvKnCgJxSwkItjq+K7wUQbSJo2t5i2i7cjUNRpT diZHTYtWEm6ZkVnOVNn0l644gYCSPSpQY0Pv3cJ8XDdd0PxaDKqPkP5IiEhWuWhGQoIVkJUM+JlA ISRCSkzDvDzNJvIfzvvmNTaLTIy1XaCt949xNgIdkb0iNHzBvvtVb1VFMxRlnX46mq97P4yWZ5mZ 8erN273d3kFMIIMTrO0NiSLDNC1AqIGn+1o54NUe81qrE3uMq2FdoAfSz7x8qql1p1q0vqYS7lJU zNVhExCiBlwUSCSCvvBfMBDdInyegUsiH618lhaCQh9BVURoX6S0cXpPui/gFfpACOSf3DhUSg2B zOXwpIZEuVv6jlSLAYlbbc3AsewLkQsmzta9WCWC+onsn3p+BJUALGiuQpyPi43vQKL20LKiaHvh Sy/7XUFTYhYU0E/DDX9t80AhRWsWxUHx5Yoh5qnhp4imn7TkiP8/BYMh2YyUfs7h5hqpmvTbQQXt 7eDmoI5oUKqiMTyz5kahTC/dbFFgUUrVFSnMAtV1tQmWBpRUyDunQmanpAevvQFogBboABSmII2g K/idgNIcoLxAVy+fnB2eHTf9+SKRiKHttdEbDKulNltQFQjsXKbdSLmEgEoMrIQmCBkElt7SsLlP QAKYRB2siIqNgMWszuABQNpYIMYKJCIA8LETQipwgrBAZnAVpD2qhLUgrURWwqqI4s3apQCrKiUb Ic7LdzpM3IG6Hur09R+ojk5AAw3FLoWLYuAnYbidOHQJ6ILzcGg7gqBsIZ0CtKQAIqvXbAvEdfRV 1BeghFtSIDUtXTrCocAqdqPR6AWOFV7v8bm3lJyREcvNC7tYpuIj1zFVRHuvld5hagNANFVURudw JkmdhEcqumhq2D07mELsvBNzPh74tIohXLNBbAo5+xuwjA4mKigkng86zl4hoJFNlLKCNHC+N3+5 R7uiAkDXRCqCRYixZpqOooio+twwX1AMxVUR09DJUuqXW7CIiBAoCyyMfhI53M7Cj1oeVAL9sKrU EB2OVILTetcqzDBpPWqXuCjURCiIq1Bos+hA4isBQginYwBBiwuiinInwIQC43DB7whcQD1BU3wM fUFddgRJIcNDusLn84CB0u5AAzscgEgBOoGW36elSXUVLkQQ9FRLREmocB8DrQlaGkD+9eh5AiRF Dq5cz3zXoeoEAPpTUUgBxseFvVr2DfFEZy7DSI5z0yh1R9ViuejknVAvt6CKX8gjjMzdGp6IbKCN AH9PSC39+eCGVILEBmPQ9CwKNyCjBSPtzMhEfcMSFckd/R0cPsUVs//vCnIgIDgoYh0kBGglFDBn eIF3P4FIjYEGmChZgIMBLIjSeUQWzAAd4Bx/YRUh5cxVUR3VLDi4WUEe5pR0VWEpFU91tFUiQUSJ IBGEFSASCEFEijFFIKscygO2moJ4ERhfR9yqNUEO1ge0G6gMTl/JRUrhF3XkcNNvgB45QgREHS4f Vgoio0aVQ8PsfxU5Ow6XXok/KEAeUUeUXp7JK3ADrkHjUbB9uRml0RuiMRHjzTxkZL06Wt75ge+E R7neCFrFASioisBXOVmBBDMRsLA/gQcikVLHCd98GNxsXpEYoFKQByiCWFSu/0I+emUzA9VVCAai A2UbnSjZJhuqTyF/qFzbDTUT0SDY80E6Fg0ckRMe1kObyOt8UkQwQ8HcQDy6IjfVlUP4R7IvAzWe NEbqKlqNbLpaoF7tJxFC0GAARCAMCAAE+uNu/l4dxS6nEv2L2EvS0XprLRr7i0LtI46gGxIYz/Z0 W0KUuVVjc1soB04SlNt6sFUDsIjq3+4K99ZaoLIK6yRezGVMjhSRTENPNgMRUoFdgV7AttyO5oFa ulg99QWQGv7synhtDaySbALEK/lVMnjS47UBvDfAn1uefJ8yIMipBRJIgyKkFEiikYZ2EsBYVeBS AwTuAJyhnIEDwOiC0CsYCimMn6G+nLjBfC7SAE6BB0g5qVj57LqR+qEmAiAKVOEuIrdIkSLPfmm6 I7x6SyCtMAev7o2pQDA+7yNtwUMCpB+wCnT2MFB5ggeajz3HvvjkJ6ICah3UECWrz8+gFvBYMiyW D4oqKjYNg0LGS90UeuSxcRqHn0AwH/7IpeLGxGkAiQUhm8xK7iIc3SMyp7grXDiFCoXVXgO/IFaA LO4HOyAr2c8LqoIzdTYEjARMhABKAGIjIIA1IiMQzZbvZMzbAGodkTQRCwrwqoZbD/FVAUDm0FAq gRigJkFZT1iOKT7qhRogXlWOSfWN0FbjAkBNoVMHVOaj0yS30uKC7b8v8lT/+LuSKcKEgX1xS1g --=-sABADkyuScs9CgvROcG/-- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2011-Feb-24 15:23 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Johannes Hirte''s message of 2011-02-23 18:02:44 -0500:> On Wednesday 23 February 2011 22:56:27 Chris Mason wrote: > > Excerpts from Zhong, Xin''s message of 2011-02-23 02:27:05 -0500: > > > In the dmesg of rc4, I can see svn hang in shrink_dellalloc and there''s > > > two flush-btrfs threads hang there too. > > > > > > Josef, it seems you are the expert in this area. Could you take a quick > > > look? Thanks! > > > > Ok, it does look like the fluhs-btrfs threads are busy trying to flush > > things. > > > > Could you please do a btrfs-show and a btrfs fi df /xxx (where xxx is > > your mount point) and send the results here? > > > > -chris > > failed to read /dev/sr0 > Label: none uuid: 00eab15f-c4cf-4403-a529-9bc11fa50167 > Total devices 1 FS bytes used 47.72GB > devid 1 size 65.69GB used 65.69GB path /dev/sda2 > > Label: none uuid: c6f4e6e6-c4ba-4394-9e9c-bbc3d0b32793 > Total devices 1 FS bytes used 9.48GB > devid 1 size 20.01GB used 20.01GB path /dev/sda1 > > Btrfs v0.19-35-g1b444cd-dirty > > and btrfs fi df on > > / > > Data: total=15.49GB, used=8.35GB > System, DUP: total=8.00MB, used=12.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=2.25GB, used=1.13GB > > /home > > Data: total=63.42GB, used=47.47GB > System: total=4.00MB, used=16.00KB > Metadata: total=2.27GB, used=251.34MB > > The bug is reproducable on both filesystems.Ok, you''ve got a good amount of metadata space free, but it is frantically trying to make room for the delayed allocation. Let me see if I can recreate this setup here. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Mitch Harder
2011-Feb-24 15:55 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
2011/2/24 Maria Wikström <maria@ponstudios.se>:> mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >> > > Haha, yes that''s very hard :) > > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the > process with ctrl+c and it disappear a few seconds later. There is no > CPU usage. Reading works because I can start htop and watch "svn info" > disappear, but everything writing to btrfs slows down to a crawl. It > takes about 1 minute to log in. So I had to put the logs on an other > partition using ext3 to get the output from sysrq+t. >I believe I''ve been experiencing this issue also. However, my problem usually results in a "No space left on device" error rather than a lock-up or crash. But I''ve bisected my issue to this patch, and my "btrfs fi show" and "btrfs fi df" looks similar to others who''ve posted to this tread with all my space being allocated, but not used. I''ve been playing around with ftrace to try to get some information on the issue. Since I''m getting a soft error, I can used a command like "echo 1 > tracing_on; emerge -1av openmotif; echo 0 > tracing_on" to end the trace as soon as the build fails. The traces are probably too large for the M/L (~275kb compressed), so I''ve put them up on my local server in case anybody can find them useful: http://dontpanic.dyndns.org/emerge-openmotif-ftrace.gz I''m still only capturing the tail end of the problem, but maybe someone will find these insightful. Let me know if you want me to increase the ftrace buffer size or insert some trace_printk debugging statements. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2011-Feb-24 16:00 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500:> 2011/2/24 Maria Wikström <maria@ponstudios.se>: > > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: > >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... > >> > > > > Haha, yes that''s very hard :) > > > > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the > > process with ctrl+c and it disappear a few seconds later. There is no > > CPU usage. Reading works because I can start htop and watch "svn info" > > disappear, but everything writing to btrfs slows down to a crawl. It > > takes about 1 minute to log in. So I had to put the logs on an other > > partition using ext3 to get the output from sysrq+t. > > > > I believe I''ve been experiencing this issue also. However, my problem > usually results in a "No space left on device" error rather than a > lock-up or crash. But I''ve bisected my issue to this patch, and my > "btrfs fi show" and "btrfs fi df" looks similar to others who''ve > posted to this tread with all my space being allocated, but not used. >Sorry, which patch did you bisect the problem down to?> I''ve been playing around with ftrace to try to get some information on > the issue. Since I''m getting a soft error, I can used a command like > "echo 1 > tracing_on; emerge -1av openmotif; echo 0 > tracing_on" to > end the trace as soon as the build fails. > > The traces are probably too large for the M/L (~275kb compressed), so > I''ve put them up on my local server in case anybody can find them > useful: > > http://dontpanic.dyndns.org/emerge-openmotif-ftrace.gz > > I''m still only capturing the tail end of the problem, but maybe > someone will find these insightful. > > Let me know if you want me to increase the ftrace buffer size or > insert some trace_printk debugging statements.Thanks, I''ll take a look. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Mitch Harder
2011-Feb-24 16:03 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason <chris.mason@oracle.com> wrote:> Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500: >> 2011/2/24 Maria Wikström <maria@ponstudios.se>: >> > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: >> >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >> >> >> > >> > Haha, yes that''s very hard :) >> > >> > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the >> > process with ctrl+c and it disappear a few seconds later. There is no >> > CPU usage. Reading works because I can start htop and watch "svn info" >> > disappear, but everything writing to btrfs slows down to a crawl. It >> > takes about 1 minute to log in. So I had to put the logs on an other >> > partition using ext3 to get the output from sysrq+t. >> > >> >> I believe I''ve been experiencing this issue also. However, my problem >> usually results in a "No space left on device" error rather than a >> lock-up or crash. But I''ve bisected my issue to this patch, and my >> "btrfs fi show" and "btrfs fi df" looks similar to others who''ve >> posted to this tread with all my space being allocated, but not used. >> > > Sorry, which patch did you bisect the problem down to? >The patch at the head of this thread: Btrfs: pwrite blocked when writing from the mmaped buffer of the same page -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2011-Feb-24 16:19 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Mitch Harder''s message of 2011-02-24 11:03:07 -0500:> On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason <chris.mason@oracle.com> wrote: > > Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500: > >> 2011/2/24 Maria Wikström <maria@ponstudios.se>: > >> > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: > >> >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... > >> >> > >> > > >> > Haha, yes that''s very hard :) > >> > > >> > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the > >> > process with ctrl+c and it disappear a few seconds later. There is no > >> > CPU usage. Reading works because I can start htop and watch "svn info" > >> > disappear, but everything writing to btrfs slows down to a crawl. It > >> > takes about 1 minute to log in. So I had to put the logs on an other > >> > partition using ext3 to get the output from sysrq+t. > >> > > >> > >> I believe I''ve been experiencing this issue also.  However, my problem > >> usually results in a "No space left on device" error rather than a > >> lock-up or crash.  But I''ve bisected my issue to this patch, and my > >> "btrfs fi show" and "btrfs fi df" looks similar to others who''ve > >> posted to this tread with all my space being allocated, but not used. > >> > > > > Sorry, which patch did you bisect the problem down to? > > > > The patch at the head of this thread: > > Btrfs: pwrite blocked when writing from the mmaped buffer of the same pageHmmm, that patch shouldn''t be changing our performance under delalloc pressure, and it really shouldn''t impact early enospc. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Mitch Harder
2011-Feb-24 16:32 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thu, Feb 24, 2011 at 10:19 AM, Chris Mason <chris.mason@oracle.com> wrote:> Excerpts from Mitch Harder''s message of 2011-02-24 11:03:07 -0500: >> On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason <chris.mason@oracle.com> wrote: >> > Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500: >> >> 2011/2/24 Maria Wikström <maria@ponstudios.se>: >> >> > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: >> >> >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >> >> >> >> >> > >> >> > Haha, yes that''s very hard :) >> >> > >> >> > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the >> >> > process with ctrl+c and it disappear a few seconds later. There is no >> >> > CPU usage. Reading works because I can start htop and watch "svn info" >> >> > disappear, but everything writing to btrfs slows down to a crawl. It >> >> > takes about 1 minute to log in. So I had to put the logs on an other >> >> > partition using ext3 to get the output from sysrq+t. >> >> > >> >> >> >> I believe I''ve been experiencing this issue also. However, my problem >> >> usually results in a "No space left on device" error rather than a >> >> lock-up or crash. But I''ve bisected my issue to this patch, and my >> >> "btrfs fi show" and "btrfs fi df" looks similar to others who''ve >> >> posted to this tread with all my space being allocated, but not used. >> >> >> > >> > Sorry, which patch did you bisect the problem down to? >> > >> >> The patch at the head of this thread: >> >> Btrfs: pwrite blocked when writing from the mmaped buffer of the same page > > Hmmm, that patch shouldn''t be changing our performance under delalloc > pressure, and it really shouldn''t impact early enospc. >I''ve bisected this issue around where this patch went into git, and I''ve also constructed a testing patch that reverts this patch, and placed it on top of the current Btrfs git sources (I understand this patch addresses a real issue, this was just for testing). It could be that this patch just "uncovers" another problem, but all my tests seem to point to this patch triggering this issue. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Piotr Szymaniak
2011-Feb-24 23:35 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Mon, Feb 21, 2011 at 09:51:11AM +0800, Zhong, Xin wrote:> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info...Hi list. I''m watching this list for a while because it seems, that I''m also affected by this bug. In the archives I found someone with Gentoo system with freezing `svn info'' (thats why I joined). Well, seems that I have same issue here. Attached latest _rc kernel sysrq+t output (first part when the `svn info'' freezed on libgcrypt, and second after ctrl+c that emerge). Seems that my backtrace is small compared to Marias. I''m missing some features in kernel to get larger backtraces? Piotr Szymaniak.> -----Original Message----- > From: Maria Wikström [mailto:maria@ponstudios.se] > Sent: Friday, February 18, 2011 7:32 PM > To: Zhong, Xin > Cc: Johannes Hirte; linux-btrfs@vger.kernel.org > Subject: RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page > > > Seems like my reply got eaten by the lists spam filter, so I resend with attachment compressed. Should have thought of that :p > > > fre 2011-02-11 klockan 12:39 +0800 skrev Zhong, Xin: > > Hi, > > > > Could you paste the output of sysrq+t here? Thanks! > > Yes, it''s in the attachment. I tried latest btrfs from git (last commit Mon Feb 14 00:45:29 2011 +0000) but it hang so bad that I couldn''t get the output from sysrq+t to hit the disk. So the output is from vanilla > 2.6.37 > > // Maria > > > -----Original Message----- > > From: Johannes Hirte [mailto:johannes.hirte@fem.tu-ilmenau.de] > > Sent: Wednesday, February 02, 2011 7:35 AM > > To: Zhong, Xin > > Cc: Maria Wikström; linux-btrfs@vger.kernel.org > > Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the > > mmaped buffer of the same page > > > > On Friday 28 January 2011 04:53:24 Zhong, Xin wrote: > > > Could you describe the steps to recreate it? > > > It will be a great help for me to look further. Thanks! > > > > It''s a little strange. I have to systems with btrfs, both > > Gentoo-based. One is affected by this bug the other is not. On the > > affected system it is enough to do a ''emerge dev-libs/libgcrypt'' that > > should normaly compile and install libgcrypt. The emerge command is > > part of portage, the package management of Gentoo. > > The strace output looks similar to the one from Maria: > > > > N?????r??y????b?X??ǧv?^?){.n?+????{?n?߲)????w*jg????????ݢj/???z?ޖ??2?ޙ????&?)ߡ?a?????G???h??j:+v???w??٥-- Druzyna futbolowa tutaj jest do niczego, ale ucze sie troche pilki noznej. Trener mowi, ze pilka nozna to futbol dla inteligentnych, a futbol to futbol dla kretynow. -- Stephen King, "The Dead Zone"
Mitch Harder
2011-Feb-25 17:11 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thu, Feb 24, 2011 at 5:14 PM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:> On Thu, Feb 24, 2011 at 10:32 AM, Mitch Harder > <mitch.harder@sabayonlinux.org> wrote: >> On Thu, Feb 24, 2011 at 10:19 AM, Chris Mason <chris.mason@oracle.com> wrote: >>> Excerpts from Mitch Harder''s message of 2011-02-24 11:03:07 -0500: >>>> On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason <chris.mason@oracle.com> wrote: >>>> > Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500: >>>> >> 2011/2/24 Maria Wikström <maria@ponstudios.se>: >>>> >> > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: >>>> >> >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >>>> >> >> >>>> >> > >>>> >> > Haha, yes that''s very hard :) >>>> >> > >>>> >> > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the >>>> >> > process with ctrl+c and it disappear a few seconds later. There is no >>>> >> > CPU usage. Reading works because I can start htop and watch "svn info" >>>> >> > disappear, but everything writing to btrfs slows down to a crawl. It >>>> >> > takes about 1 minute to log in. So I had to put the logs on an other >>>> >> > partition using ext3 to get the output from sysrq+t. >>>> >> > >>>> >> >>>> >> I believe I''ve been experiencing this issue also. However, my problem >>>> >> usually results in a "No space left on device" error rather than a >>>> >> lock-up or crash. But I''ve bisected my issue to this patch, and my >>>> >> "btrfs fi show" and "btrfs fi df" looks similar to others who''ve >>>> >> posted to this tread with all my space being allocated, but not used. >>>> >> >>>> > >>>> > Sorry, which patch did you bisect the problem down to? >>>> > >>>> >>>> The patch at the head of this thread: >>>> >>>> Btrfs: pwrite blocked when writing from the mmaped buffer of the same page >>> >>> Hmmm, that patch shouldn''t be changing our performance under delalloc >>> pressure, and it really shouldn''t impact early enospc. >>> >> >> I''ve bisected this issue around where this patch went into git, and >> I''ve also constructed a testing patch that reverts this patch, and >> placed it on top of the current Btrfs git sources (I understand this >> patch addresses a real issue, this was just for testing). >> >> It could be that this patch just "uncovers" another problem, but all >> my tests seem to point to this patch triggering this issue. >> > > I don''t belief the previous ftrace I supplied had a large enough scope > to capture the issue. > > I''ve expanded my ftrace buffer, and filtered out everything but btrfs* > function calls ("# echo btrfs* > > /sys/kernel/debug/tracing/set_ftrace_filter"). > > In this trace, I see btrfs spending a great deal of time in a while > loop (while (iov_iter_count(&i) > 0) {)) in the btrfs_file_aio_write() > function in file.c without exiting the function. > > I''m going to try to inject some debugging trace_printk() statements to > find if that portion of code is proceeding normally with my test case. > > I''ve put my expanded trace up on my local server, but my upload > bandwidth is pretty sad, and it may take a few minutes to transfer > even though it''s only a 6MB file. > > http://dontpanic.dyndns.org/trace-openmotif-btrfs-v3.gz >Apologies for only hitting "Reply" instead of "Reply-All" on my last message. I''ve inserted additional trace_printk() to the btrfs_file_aio_write() and btrfs_copy_from_user() function in file.c in order to characterize the problem I''ve been encountering. I can see btrfs getting stuck in a loop in the "while (iov_iter_count(&i) > 0) {}" portion of the btrfs_file_aio_write() function. The loop is more-or-less following this process (from within the "while (iov_iter_count(&i) > 0) {}" loop): (1) Reserve some space with btrfs_delalloc_reserve_space() (2) Prepare the reserved space with prepare_pages() (3) Call btrfs_copy_from_user() to copy to the prepared space. -------------> From btrfs_copy_from_user() (4) ........Try to copy with copied = iov_iter_copy_from_user_atomic() (5) ........The above operation results with copied == 0. Break and return with a return value of 0 bytes copied. (6) There is no special handling for copied == 0 in the "while (iov_iter_count(&i) > 0) {}" loop, so it loops back around, reserves some more space, and tries again. If I look back at how the code was set up before the patch at the head of this thread was applied (Btrfs: pwrite blocked when writing from the mmaped buffer of the same page), the btrfs_copy_from_user() function had some handling for "copied == 0" that would change the scope of the amount to write, and loop back to try the write again. I attempted to construct a patch that just reverted the handling for "copied == 0" in btrfs_copy_from_user(), however, that just resulted in my computer locking up when it reached the point where it was previously beginning to allocate disk space. So, I apologize for not having a patch to address the issue I''m seeing, but I hope I''ve added some insight. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Mitch Harder
2011-Feb-25 18:43 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Fri, Feb 25, 2011 at 11:11 AM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:> On Thu, Feb 24, 2011 at 5:14 PM, Mitch Harder > <mitch.harder@sabayonlinux.org> wrote: >> On Thu, Feb 24, 2011 at 10:32 AM, Mitch Harder >> <mitch.harder@sabayonlinux.org> wrote: >>> On Thu, Feb 24, 2011 at 10:19 AM, Chris Mason <chris.mason@oracle.com> wrote: >>>> Excerpts from Mitch Harder''s message of 2011-02-24 11:03:07 -0500: >>>>> On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason <chris.mason@oracle.com> wrote: >>>>> > Excerpts from Mitch Harder''s message of 2011-02-24 10:55:15 -0500: >>>>> >> 2011/2/24 Maria Wikström <maria@ponstudios.se>: >>>>> >> > mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: >>>>> >> >> The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there''s no debug info... >>>>> >> >> >>>>> >> > >>>>> >> > Haha, yes that''s very hard :) >>>>> >> > >>>>> >> > 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the >>>>> >> > process with ctrl+c and it disappear a few seconds later. There is no >>>>> >> > CPU usage. Reading works because I can start htop and watch "svn info" >>>>> >> > disappear, but everything writing to btrfs slows down to a crawl. It >>>>> >> > takes about 1 minute to log in. So I had to put the logs on an other >>>>> >> > partition using ext3 to get the output from sysrq+t. >>>>> >> > >>>>> >> >>>>> >> I believe I''ve been experiencing this issue also. However, my problem >>>>> >> usually results in a "No space left on device" error rather than a >>>>> >> lock-up or crash. But I''ve bisected my issue to this patch, and my >>>>> >> "btrfs fi show" and "btrfs fi df" looks similar to others who''ve >>>>> >> posted to this tread with all my space being allocated, but not used. >>>>> >> >>>>> > >>>>> > Sorry, which patch did you bisect the problem down to? >>>>> > >>>>> >>>>> The patch at the head of this thread: >>>>> >>>>> Btrfs: pwrite blocked when writing from the mmaped buffer of the same page >>>> >>>> Hmmm, that patch shouldn''t be changing our performance under delalloc >>>> pressure, and it really shouldn''t impact early enospc. >>>> >>> >>> I''ve bisected this issue around where this patch went into git, and >>> I''ve also constructed a testing patch that reverts this patch, and >>> placed it on top of the current Btrfs git sources (I understand this >>> patch addresses a real issue, this was just for testing). >>> >>> It could be that this patch just "uncovers" another problem, but all >>> my tests seem to point to this patch triggering this issue. >>> >> >> I don''t belief the previous ftrace I supplied had a large enough scope >> to capture the issue. >> >> I''ve expanded my ftrace buffer, and filtered out everything but btrfs* >> function calls ("# echo btrfs* > >> /sys/kernel/debug/tracing/set_ftrace_filter"). >> >> In this trace, I see btrfs spending a great deal of time in a while >> loop (while (iov_iter_count(&i) > 0) {)) in the btrfs_file_aio_write() >> function in file.c without exiting the function. >> >> I''m going to try to inject some debugging trace_printk() statements to >> find if that portion of code is proceeding normally with my test case. >> >> I''ve put my expanded trace up on my local server, but my upload >> bandwidth is pretty sad, and it may take a few minutes to transfer >> even though it''s only a 6MB file. >> >> http://dontpanic.dyndns.org/trace-openmotif-btrfs-v3.gz >> > > Apologies for only hitting "Reply" instead of "Reply-All" on my last message. > > I''ve inserted additional trace_printk() to the btrfs_file_aio_write() > and btrfs_copy_from_user() function in file.c in order to characterize > the problem I''ve been encountering. > > I can see btrfs getting stuck in a loop in the "while > (iov_iter_count(&i) > 0) {}" portion of the btrfs_file_aio_write() > function. > > The loop is more-or-less following this process (from within the > "while (iov_iter_count(&i) > 0) {}" loop): > > (1) Reserve some space with btrfs_delalloc_reserve_space() > (2) Prepare the reserved space with prepare_pages() > (3) Call btrfs_copy_from_user() to copy to the prepared space. > -------------> From btrfs_copy_from_user() > (4) ........Try to copy with copied = iov_iter_copy_from_user_atomic() > (5) ........The above operation results with copied == 0. Break and > return with a return value of 0 bytes copied. > (6) There is no special handling for copied == 0 in the "while > (iov_iter_count(&i) > 0) {}" loop, so it loops back around, reserves > some more space, and tries again. > > If I look back at how the code was set up before the patch at the head > of this thread was applied (Btrfs: pwrite blocked when writing from > the mmaped buffer of the same page), the btrfs_copy_from_user() > function had some handling for "copied == 0" that would change the > scope of the amount to write, and loop back to try the write again. > > I attempted to construct a patch that just reverted the handling for > "copied == 0" in btrfs_copy_from_user(), however, that just resulted > in my computer locking up when it reached the point where it was > previously beginning to allocate disk space. > > So, I apologize for not having a patch to address the issue I''m > seeing, but I hope I''ve added some insight. >Some clarification on my previous message... After looking at my ftrace log more closely, I can see where Btrfs is trying to release the allocated pages. However, the calculation for the number of dirty_pages is equal to 1 when "copied == 0". So I''m seeing at least two problems: (1) It keeps looping when "copied == 0". (2) One dirty page is not being released on every loop even though "copied == 0" (at least this problem keeps it from being an infinite loop by eventually exhausting reserveable space on the disk). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason
2011-Feb-25 19:19 UTC
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500:> > The loop is more-or-less following this process (from within the > > "while (iov_iter_count(&i) > 0) {}" loop): > > > > (1) Reserve some space with btrfs_delalloc_reserve_space() > > (2) Prepare the reserved space with prepare_pages() > > (3) Call btrfs_copy_from_user() to copy to the prepared space. > > -------------> From btrfs_copy_from_user() > > (4) ........Try to copy with copied = iov_iter_copy_from_user_atomic() > > (5) ........The above operation results with copied == 0. Break and > > return with a return value of 0 bytes copied. > > (6) There is no special handling for copied == 0 in the "while > > (iov_iter_count(&i) > 0) {}" loop, so it loops back around, reserves > > some more space, and tries again. > > > > If I look back at how the code was set up before the patch at the head > > of this thread was applied (Btrfs: pwrite blocked when writing from > > the mmaped buffer of the same page), the btrfs_copy_from_user() > > function had some handling for "copied == 0" that would change the > > scope of the amount to write, and loop back to try the write again. > > > > I attempted to construct a patch that just reverted the handling for > > "copied == 0" in btrfs_copy_from_user(), however, that just resulted > > in my computer locking up when it reached the point where it was > > previously beginning to allocate disk space. > > > > So, I apologize for not having a patch to address the issue I''m > > seeing, but I hope I''ve added some insight. > > > > Some clarification on my previous message... > > After looking at my ftrace log more closely, I can see where Btrfs is > trying to release the allocated pages. However, the calculation for > the number of dirty_pages is equal to 1 when "copied == 0". > > So I''m seeing at least two problems: > (1) It keeps looping when "copied == 0". > (2) One dirty page is not being released on every loop even though > "copied == 0" (at least this problem keeps it from being an infinite > loop by eventually exhausting reserveable space on the disk).Very nice, I think you''re exactly right. I should be able to reproduce this now, thanks! -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500:> Some clarification on my previous message... > > After looking at my ftrace log more closely, I can see where Btrfs is > trying to release the allocated pages. However, the calculation for > the number of dirty_pages is equal to 1 when "copied == 0". > > So I''m seeing at least two problems: > (1) It keeps looping when "copied == 0". > (2) One dirty page is not being released on every loop even though > "copied == 0" (at least this problem keeps it from being an infinite > loop by eventually exhausting reserveable space on the disk).Hi everyone, There are actually tow bugs here. First the one that Mitch hit, and a second one that still results in bad file_write results with my debugging hunks (the first two hunks below) in place. My patch fixes Mitch''s bug by checking for copied == 0 after btrfs_copy_from_user and going the correct delalloc accounting. This one looks solved, but you''ll notice the patch is bigger. First, I add some random failures to btrfs_copy_from_user() by failing everyone once and a while. This was much more reliable than trying to use memory pressure than making copy_from_user fail. If copy_from_user fails and we partially update a page, we end up with a page that may go away due to memory pressure. But, btrfs_file_write assumes that only the first and last page may have good data that needs to be read off the disk. This patch ditches that code and puts it into prepare_pages instead. But I''m still having some errors during long stress.sh runs. Ideas are more than welcome, hopefully some other timezones will kick in ideas while I sleep. diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 7084140..89a6a26 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -54,6 +54,13 @@ static noinline int btrfs_copy_from_user(loff_t pos, int num_pages, int offset = pos & (PAGE_CACHE_SIZE - 1); int total_copied = 0; + if ((jiffies % 10) == 0) + return 0; + + if ((jiffies % 25) == 0) { + write_bytes /= 2; + } + while (write_bytes > 0) { size_t count = min_t(size_t, PAGE_CACHE_SIZE - offset, write_bytes); @@ -763,6 +770,26 @@ out: } /* + * on error we return an unlocked page and the error value + * on success we return a locked page and 0 + */ +static int prepare_uptodate_page(struct page *page, u64 pos) +{ + int ret = 0; + if ((pos & (PAGE_CACHE_SIZE - 1)) && !PageUptodate(page)) { + ret = btrfs_readpage(NULL, page); + if (ret) + return ret; + lock_page(page); + if (!PageUptodate(page)) { + unlock_page(page); + return -EIO; + } + } + return 0; +} + +/* * this gets pages into the page cache and locks them down, it also properly * waits for data=ordered extents to finish before allowing the pages to be * modified. @@ -777,6 +804,7 @@ static noinline int prepare_pages(struct btrfs_root *root, struct file *file, unsigned long index = pos >> PAGE_CACHE_SHIFT; struct inode *inode = fdentry(file)->d_inode; int err = 0; + int faili = 0; u64 start_pos; u64 last_pos; @@ -794,15 +822,24 @@ again: for (i = 0; i < num_pages; i++) { pages[i] = grab_cache_page(inode->i_mapping, index + i); if (!pages[i]) { - int c; - for (c = i - 1; c >= 0; c--) { - unlock_page(pages[c]); - page_cache_release(pages[c]); - } - return -ENOMEM; + faili = i - 1; + err = -ENOMEM; + goto fail; + } + + if (i == 0) + err = prepare_uptodate_page(pages[i], pos); + else if (i == num_pages - 1) + err = prepare_uptodate_page(pages[i], + pos + write_bytes); + if (err) { + page_cache_release(pages[i]); + faili = i - 1; + goto fail; } wait_on_page_writeback(pages[i]); } + err = 0; if (start_pos < inode->i_size) { struct btrfs_ordered_extent *ordered; lock_extent_bits(&BTRFS_I(inode)->io_tree, @@ -842,6 +879,14 @@ again: WARN_ON(!PageLocked(pages[i])); } return 0; +fail: + while (faili >= 0) { + unlock_page(pages[faili]); + page_cache_release(pages[faili]); + faili--; + } + return err; + } static ssize_t btrfs_file_aio_write(struct kiocb *iocb, @@ -851,7 +896,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, struct file *file = iocb->ki_filp; struct inode *inode = fdentry(file)->d_inode; struct btrfs_root *root = BTRFS_I(inode)->root; - struct page *pinned[2]; struct page **pages = NULL; struct iov_iter i; loff_t *ppos = &iocb->ki_pos; @@ -872,9 +916,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || (file->f_flags & O_DIRECT)); - pinned[0] = NULL; - pinned[1] = NULL; - start_pos = pos; vfs_check_frozen(inode->i_sb, SB_FREEZE_WRITE); @@ -962,32 +1003,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, first_index = pos >> PAGE_CACHE_SHIFT; last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; - /* - * there are lots of better ways to do this, but this code - * makes sure the first and last page in the file range are - * up to date and ready for cow - */ - if ((pos & (PAGE_CACHE_SIZE - 1))) { - pinned[0] = grab_cache_page(inode->i_mapping, first_index); - if (!PageUptodate(pinned[0])) { - ret = btrfs_readpage(NULL, pinned[0]); - BUG_ON(ret); - wait_on_page_locked(pinned[0]); - } else { - unlock_page(pinned[0]); - } - } - if ((pos + iov_iter_count(&i)) & (PAGE_CACHE_SIZE - 1)) { - pinned[1] = grab_cache_page(inode->i_mapping, last_index); - if (!PageUptodate(pinned[1])) { - ret = btrfs_readpage(NULL, pinned[1]); - BUG_ON(ret); - wait_on_page_locked(pinned[1]); - } else { - unlock_page(pinned[1]); - } - } - while (iov_iter_count(&i) > 0) { size_t offset = pos & (PAGE_CACHE_SIZE - 1); size_t write_bytes = min(iov_iter_count(&i), @@ -1024,8 +1039,12 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, copied = btrfs_copy_from_user(pos, num_pages, write_bytes, pages, &i); - dirty_pages = (copied + offset + PAGE_CACHE_SIZE - 1) >> - PAGE_CACHE_SHIFT; + if (copied == 0) + dirty_pages = 0; + else + dirty_pages = (copied + offset + + PAGE_CACHE_SIZE - 1) >> + PAGE_CACHE_SHIFT; if (num_pages > dirty_pages) { if (copied > 0) @@ -1069,10 +1088,6 @@ out: err = ret; kfree(pages); - if (pinned[0]) - page_cache_release(pinned[0]); - if (pinned[1]) - page_cache_release(pinned[1]); *ppos = pos; /* -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
One possible issue I can see is in the random failure case #2 that copy_from_user only process half of the data. For example, if it write a 4k aligned page and copy_from_user only write 2k. Then it will not call btrfs_delalloc_release_space since num_pages and dirty_pages are both 1. In the next round, it write another 2k and btrfs_delalloc_reserve_space is called twice for the same page. Is it a problem? Thanks! -----Original Message----- From: Chris Mason [mailto:chris.mason@oracle.com] Sent: Monday, February 28, 2011 9:46 AM To: Mitch Harder Cc: Maria Wikström; Zhong, Xin; Johannes Hirte; linux-btrfs@vger.kernel.org Subject: [PATCH] btrfs file write debugging patch Excerpts from Mitch Harder's message of 2011-02-25 13:43:37 -0500:> Some clarification on my previous message... > > After looking at my ftrace log more closely, I can see where Btrfs is > trying to release the allocated pages. However, the calculation for > the number of dirty_pages is equal to 1 when "copied == 0". > > So I'm seeing at least two problems: > (1) It keeps looping when "copied == 0". > (2) One dirty page is not being released on every loop even though > "copied == 0" (at least this problem keeps it from being an infinite > loop by eventually exhausting reserveable space on the disk).Hi everyone, There are actually tow bugs here. First the one that Mitch hit, and a second one that still results in bad file_write results with my debugging hunks (the first two hunks below) in place. My patch fixes Mitch's bug by checking for copied == 0 after btrfs_copy_from_user and going the correct delalloc accounting. This one looks solved, but you'll notice the patch is bigger. First, I add some random failures to btrfs_copy_from_user() by failing everyone once and a while. This was much more reliable than trying to use memory pressure than making copy_from_user fail. If copy_from_user fails and we partially update a page, we end up with a page that may go away due to memory pressure. But, btrfs_file_write assumes that only the first and last page may have good data that needs to be read off the disk. This patch ditches that code and puts it into prepare_pages instead. But I'm still having some errors during long stress.sh runs. Ideas are more than welcome, hopefully some other timezones will kick in ideas while I sleep. diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 7084140..89a6a26 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -54,6 +54,13 @@ static noinline int btrfs_copy_from_user(loff_t pos, int num_pages, int offset = pos & (PAGE_CACHE_SIZE - 1); int total_copied = 0; + if ((jiffies % 10) == 0) + return 0; + + if ((jiffies % 25) == 0) { + write_bytes /= 2; + } + while (write_bytes > 0) { size_t count = min_t(size_t, PAGE_CACHE_SIZE - offset, write_bytes); @@ -763,6 +770,26 @@ out: } /* + * on error we return an unlocked page and the error value + * on success we return a locked page and 0 + */ +static int prepare_uptodate_page(struct page *page, u64 pos) +{ + int ret = 0; + if ((pos & (PAGE_CACHE_SIZE - 1)) && !PageUptodate(page)) { + ret = btrfs_readpage(NULL, page); + if (ret) + return ret; + lock_page(page); + if (!PageUptodate(page)) { + unlock_page(page); + return -EIO; + } + } + return 0; +} + +/* * this gets pages into the page cache and locks them down, it also properly * waits for data=ordered extents to finish before allowing the pages to be * modified. @@ -777,6 +804,7 @@ static noinline int prepare_pages(struct btrfs_root *root, struct file *file, unsigned long index = pos >> PAGE_CACHE_SHIFT; struct inode *inode = fdentry(file)->d_inode; int err = 0; + int faili = 0; u64 start_pos; u64 last_pos; @@ -794,15 +822,24 @@ again: for (i = 0; i < num_pages; i++) { pages[i] = grab_cache_page(inode->i_mapping, index + i); if (!pages[i]) { - int c; - for (c = i - 1; c >= 0; c--) { - unlock_page(pages[c]); - page_cache_release(pages[c]); - } - return -ENOMEM; + faili = i - 1; + err = -ENOMEM; + goto fail; + } + + if (i == 0) + err = prepare_uptodate_page(pages[i], pos); + else if (i == num_pages - 1) + err = prepare_uptodate_page(pages[i], + pos + write_bytes); + if (err) { + page_cache_release(pages[i]); + faili = i - 1; + goto fail; } wait_on_page_writeback(pages[i]); } + err = 0; if (start_pos < inode->i_size) { struct btrfs_ordered_extent *ordered; lock_extent_bits(&BTRFS_I(inode)->io_tree, @@ -842,6 +879,14 @@ again: WARN_ON(!PageLocked(pages[i])); } return 0; +fail: + while (faili >= 0) { + unlock_page(pages[faili]); + page_cache_release(pages[faili]); + faili--; + } + return err; + } static ssize_t btrfs_file_aio_write(struct kiocb *iocb, @@ -851,7 +896,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, struct file *file = iocb->ki_filp; struct inode *inode = fdentry(file)->d_inode; struct btrfs_root *root = BTRFS_I(inode)->root; - struct page *pinned[2]; struct page **pages = NULL; struct iov_iter i; loff_t *ppos = &iocb->ki_pos; @@ -872,9 +916,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || (file->f_flags & O_DIRECT)); - pinned[0] = NULL; - pinned[1] = NULL; - start_pos = pos; vfs_check_frozen(inode->i_sb, SB_FREEZE_WRITE); @@ -962,32 +1003,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, first_index = pos >> PAGE_CACHE_SHIFT; last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; - /* - * there are lots of better ways to do this, but this code - * makes sure the first and last page in the file range are - * up to date and ready for cow - */ - if ((pos & (PAGE_CACHE_SIZE - 1))) { - pinned[0] = grab_cache_page(inode->i_mapping, first_index); - if (!PageUptodate(pinned[0])) { - ret = btrfs_readpage(NULL, pinned[0]); - BUG_ON(ret); - wait_on_page_locked(pinned[0]); - } else { - unlock_page(pinned[0]); - } - } - if ((pos + iov_iter_count(&i)) & (PAGE_CACHE_SIZE - 1)) { - pinned[1] = grab_cache_page(inode->i_mapping, last_index); - if (!PageUptodate(pinned[1])) { - ret = btrfs_readpage(NULL, pinned[1]); - BUG_ON(ret); - wait_on_page_locked(pinned[1]); - } else { - unlock_page(pinned[1]); - } - } - while (iov_iter_count(&i) > 0) { size_t offset = pos & (PAGE_CACHE_SIZE - 1); size_t write_bytes = min(iov_iter_count(&i), @@ -1024,8 +1039,12 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, copied = btrfs_copy_from_user(pos, num_pages, write_bytes, pages, &i); - dirty_pages = (copied + offset + PAGE_CACHE_SIZE - 1) >> - PAGE_CACHE_SHIFT; + if (copied == 0) + dirty_pages = 0; + else + dirty_pages = (copied + offset + + PAGE_CACHE_SIZE - 1) >> + PAGE_CACHE_SHIFT; if (num_pages > dirty_pages) { if (copied > 0) @@ -1069,10 +1088,6 @@ out: err = ret; kfree(pages); - if (pinned[0]) - page_cache_release(pinned[0]); - if (pinned[1]) - page_cache_release(pinned[1]); *ppos = pos; /*
On Monday 28 February 2011 02:46:05 Chris Mason wrote:> Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: > > Some clarification on my previous message... > > > > After looking at my ftrace log more closely, I can see where Btrfs is > > trying to release the allocated pages. However, the calculation for > > the number of dirty_pages is equal to 1 when "copied == 0". > > > > So I''m seeing at least two problems: > > (1) It keeps looping when "copied == 0". > > (2) One dirty page is not being released on every loop even though > > "copied == 0" (at least this problem keeps it from being an infinite > > loop by eventually exhausting reserveable space on the disk). > > Hi everyone, > > There are actually tow bugs here. First the one that Mitch hit, and a > second one that still results in bad file_write results with my > debugging hunks (the first two hunks below) in place. > > My patch fixes Mitch''s bug by checking for copied == 0 after > btrfs_copy_from_user and going the correct delalloc accounting. This > one looks solved, but you''ll notice the patch is bigger. > > First, I add some random failures to btrfs_copy_from_user() by failing > everyone once and a while. This was much more reliable than trying to > use memory pressure than making copy_from_user fail. > > If copy_from_user fails and we partially update a page, we end up with a > page that may go away due to memory pressure. But, btrfs_file_write > assumes that only the first and last page may have good data that needs > to be read off the disk. > > This patch ditches that code and puts it into prepare_pages instead. > But I''m still having some errors during long stress.sh runs. Ideas are > more than welcome, hopefully some other timezones will kick in ideas > while I sleep.At least it doesn''t fix the emerge-problem for me. The behavior is now the same as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no further interaction to get the emerge-process hang with a svn-process consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the spawned svn-process stays and it needs a reboot to get rid of it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Excerpts from Johannes Hirte''s message of 2011-02-28 05:13:59 -0500:> On Monday 28 February 2011 02:46:05 Chris Mason wrote: > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: > > > Some clarification on my previous message... > > > > > > After looking at my ftrace log more closely, I can see where Btrfs is > > > trying to release the allocated pages. However, the calculation for > > > the number of dirty_pages is equal to 1 when "copied == 0". > > > > > > So I''m seeing at least two problems: > > > (1) It keeps looping when "copied == 0". > > > (2) One dirty page is not being released on every loop even though > > > "copied == 0" (at least this problem keeps it from being an infinite > > > loop by eventually exhausting reserveable space on the disk). > > > > Hi everyone, > > > > There are actually tow bugs here. First the one that Mitch hit, and a > > second one that still results in bad file_write results with my > > debugging hunks (the first two hunks below) in place. > > > > My patch fixes Mitch''s bug by checking for copied == 0 after > > btrfs_copy_from_user and going the correct delalloc accounting. This > > one looks solved, but you''ll notice the patch is bigger. > > > > First, I add some random failures to btrfs_copy_from_user() by failing > > everyone once and a while. This was much more reliable than trying to > > use memory pressure than making copy_from_user fail. > > > > If copy_from_user fails and we partially update a page, we end up with a > > page that may go away due to memory pressure. But, btrfs_file_write > > assumes that only the first and last page may have good data that needs > > to be read off the disk. > > > > This patch ditches that code and puts it into prepare_pages instead. > > But I''m still having some errors during long stress.sh runs. Ideas are > > more than welcome, hopefully some other timezones will kick in ideas > > while I sleep. > > At least it doesn''t fix the emerge-problem for me. The behavior is now the same > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no > further interaction to get the emerge-process hang with a svn-process > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the > spawned svn-process stays and it needs a reboot to get rid of it.I think your problem really is more enospc related. Still working on that as well. But please don''t try the patch without removing the debugging hunk at the top (anything that mentions jiffies). -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Excerpts from Zhong, Xin''s message of 2011-02-28 03:56:40 -0500:> One possible issue I can see is in the random failure case #2 that copy_from_user only process half of the data. > > For example, if it write a 4k aligned page and copy_from_user only write 2k. Then it will not call btrfs_delalloc_release_space since num_pages and dirty_pages are both 1. > In the next round, it write another 2k and btrfs_delalloc_reserve_space is called twice for the same page. > > Is it a problem? Thanks!It should be the correct answer. The result of the short copy_from_user should be exactly the same as two write calls where one does 2K and the other does another 2K. Either way, it shouldn''t result in incorrect bytes in the file, which is still happening for me with the debugging hunks in place. -chris> > -----Original Message----- > From: Chris Mason [mailto:chris.mason@oracle.com] > Sent: Monday, February 28, 2011 9:46 AM > To: Mitch Harder > Cc: Maria Wikström; Zhong, Xin; Johannes Hirte; linux-btrfs@vger.kernel.org > Subject: [PATCH] btrfs file write debugging patch > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: > > Some clarification on my previous message... > > > > After looking at my ftrace log more closely, I can see where Btrfs is > > trying to release the allocated pages. However, the calculation for > > the number of dirty_pages is equal to 1 when "copied == 0". > > > > So I''m seeing at least two problems: > > (1) It keeps looping when "copied == 0". > > (2) One dirty page is not being released on every loop even though > > "copied == 0" (at least this problem keeps it from being an infinite > > loop by eventually exhausting reserveable space on the disk). > > Hi everyone, > > There are actually tow bugs here. First the one that Mitch hit, and a > second one that still results in bad file_write results with my > debugging hunks (the first two hunks below) in place. > > My patch fixes Mitch''s bug by checking for copied == 0 after > btrfs_copy_from_user and going the correct delalloc accounting. This > one looks solved, but you''ll notice the patch is bigger. > > First, I add some random failures to btrfs_copy_from_user() by failing > everyone once and a while. This was much more reliable than trying to > use memory pressure than making copy_from_user fail. > > If copy_from_user fails and we partially update a page, we end up with a > page that may go away due to memory pressure. But, btrfs_file_write > assumes that only the first and last page may have good data that needs > to be read off the disk. > > This patch ditches that code and puts it into prepare_pages instead. > But I''m still having some errors during long stress.sh runs. Ideas are > more than welcome, hopefully some other timezones will kick in ideas > while I sleep. > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c > index 7084140..89a6a26 100644 > --- a/fs/btrfs/file.c > +++ b/fs/btrfs/file.c > @@ -54,6 +54,13 @@ static noinline int btrfs_copy_from_user(loff_t pos, int num_pages, > int offset = pos & (PAGE_CACHE_SIZE - 1); > int total_copied = 0; > > + if ((jiffies % 10) == 0) > + return 0; > + > + if ((jiffies % 25) == 0) { > + write_bytes /= 2; > + } > + > while (write_bytes > 0) { > size_t count = min_t(size_t, > PAGE_CACHE_SIZE - offset, write_bytes); > @@ -763,6 +770,26 @@ out: > } > > /* > + * on error we return an unlocked page and the error value > + * on success we return a locked page and 0 > + */ > +static int prepare_uptodate_page(struct page *page, u64 pos) > +{ > + int ret = 0; > + if ((pos & (PAGE_CACHE_SIZE - 1)) && !PageUptodate(page)) { > + ret = btrfs_readpage(NULL, page); > + if (ret) > + return ret; > + lock_page(page); > + if (!PageUptodate(page)) { > + unlock_page(page); > + return -EIO; > + } > + } > + return 0; > +} > + > +/* > * this gets pages into the page cache and locks them down, it also properly > * waits for data=ordered extents to finish before allowing the pages to be > * modified. > @@ -777,6 +804,7 @@ static noinline int prepare_pages(struct btrfs_root *root, struct file *file, > unsigned long index = pos >> PAGE_CACHE_SHIFT; > struct inode *inode = fdentry(file)->d_inode; > int err = 0; > + int faili = 0; > u64 start_pos; > u64 last_pos; > > @@ -794,15 +822,24 @@ again: > for (i = 0; i < num_pages; i++) { > pages[i] = grab_cache_page(inode->i_mapping, index + i); > if (!pages[i]) { > - int c; > - for (c = i - 1; c >= 0; c--) { > - unlock_page(pages[c]); > - page_cache_release(pages[c]); > - } > - return -ENOMEM; > + faili = i - 1; > + err = -ENOMEM; > + goto fail; > + } > + > + if (i == 0) > + err = prepare_uptodate_page(pages[i], pos); > + else if (i == num_pages - 1) > + err = prepare_uptodate_page(pages[i], > + pos + write_bytes); > + if (err) { > + page_cache_release(pages[i]); > + faili = i - 1; > + goto fail; > } > wait_on_page_writeback(pages[i]); > } > + err = 0; > if (start_pos < inode->i_size) { > struct btrfs_ordered_extent *ordered; > lock_extent_bits(&BTRFS_I(inode)->io_tree, > @@ -842,6 +879,14 @@ again: > WARN_ON(!PageLocked(pages[i])); > } > return 0; > +fail: > + while (faili >= 0) { > + unlock_page(pages[faili]); > + page_cache_release(pages[faili]); > + faili--; > + } > + return err; > + > } > > static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > @@ -851,7 +896,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > struct file *file = iocb->ki_filp; > struct inode *inode = fdentry(file)->d_inode; > struct btrfs_root *root = BTRFS_I(inode)->root; > - struct page *pinned[2]; > struct page **pages = NULL; > struct iov_iter i; > loff_t *ppos = &iocb->ki_pos; > @@ -872,9 +916,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || > (file->f_flags & O_DIRECT)); > > - pinned[0] = NULL; > - pinned[1] = NULL; > - > start_pos = pos; > > vfs_check_frozen(inode->i_sb, SB_FREEZE_WRITE); > @@ -962,32 +1003,6 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > first_index = pos >> PAGE_CACHE_SHIFT; > last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; > > - /* > - * there are lots of better ways to do this, but this code > - * makes sure the first and last page in the file range are > - * up to date and ready for cow > - */ > - if ((pos & (PAGE_CACHE_SIZE - 1))) { > - pinned[0] = grab_cache_page(inode->i_mapping, first_index); > - if (!PageUptodate(pinned[0])) { > - ret = btrfs_readpage(NULL, pinned[0]); > - BUG_ON(ret); > - wait_on_page_locked(pinned[0]); > - } else { > - unlock_page(pinned[0]); > - } > - } > - if ((pos + iov_iter_count(&i)) & (PAGE_CACHE_SIZE - 1)) { > - pinned[1] = grab_cache_page(inode->i_mapping, last_index); > - if (!PageUptodate(pinned[1])) { > - ret = btrfs_readpage(NULL, pinned[1]); > - BUG_ON(ret); > - wait_on_page_locked(pinned[1]); > - } else { > - unlock_page(pinned[1]); > - } > - } > - > while (iov_iter_count(&i) > 0) { > size_t offset = pos & (PAGE_CACHE_SIZE - 1); > size_t write_bytes = min(iov_iter_count(&i), > @@ -1024,8 +1039,12 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > > copied = btrfs_copy_from_user(pos, num_pages, > write_bytes, pages, &i); > - dirty_pages = (copied + offset + PAGE_CACHE_SIZE - 1) >> > - PAGE_CACHE_SHIFT; > + if (copied == 0) > + dirty_pages = 0; > + else > + dirty_pages = (copied + offset + > + PAGE_CACHE_SIZE - 1) >> > + PAGE_CACHE_SHIFT; > > if (num_pages > dirty_pages) { > if (copied > 0) > @@ -1069,10 +1088,6 @@ out: > err = ret; > > kfree(pages); > - if (pinned[0]) > - page_cache_release(pinned[0]); > - if (pinned[1]) > - page_cache_release(pinned[1]); > *ppos = pos; > > /*-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote:> On Monday 28 February 2011 02:46:05 Chris Mason wrote: > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: > > > Some clarification on my previous message... > > > > > > After looking at my ftrace log more closely, I can see where Btrfs is > > > trying to release the allocated pages. However, the calculation for > > > the number of dirty_pages is equal to 1 when "copied == 0". > > > > > > So I''m seeing at least two problems: > > > (1) It keeps looping when "copied == 0". > > > (2) One dirty page is not being released on every loop even though > > > "copied == 0" (at least this problem keeps it from being an infinite > > > loop by eventually exhausting reserveable space on the disk). > > > > Hi everyone, > > > > There are actually tow bugs here. First the one that Mitch hit, and a > > second one that still results in bad file_write results with my > > debugging hunks (the first two hunks below) in place. > > > > My patch fixes Mitch''s bug by checking for copied == 0 after > > btrfs_copy_from_user and going the correct delalloc accounting. This > > one looks solved, but you''ll notice the patch is bigger. > > > > First, I add some random failures to btrfs_copy_from_user() by failing > > everyone once and a while. This was much more reliable than trying to > > use memory pressure than making copy_from_user fail. > > > > If copy_from_user fails and we partially update a page, we end up with a > > page that may go away due to memory pressure. But, btrfs_file_write > > assumes that only the first and last page may have good data that needs > > to be read off the disk. > > > > This patch ditches that code and puts it into prepare_pages instead. > > But I''m still having some errors during long stress.sh runs. Ideas are > > more than welcome, hopefully some other timezones will kick in ideas > > while I sleep. > > At least it doesn''t fix the emerge-problem for me. The behavior is now the same > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no > further interaction to get the emerge-process hang with a svn-process > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the > spawned svn-process stays and it needs a reboot to get rid of it.Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s looping? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik:> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote: > > On Monday 28 February 2011 02:46:05 Chris Mason wrote: > > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: > > > > Some clarification on my previous message... > > > > > > > > After looking at my ftrace log more closely, I can see where Btrfs is > > > > trying to release the allocated pages. However, the calculation for > > > > the number of dirty_pages is equal to 1 when "copied == 0". > > > > > > > > So I''m seeing at least two problems: > > > > (1) It keeps looping when "copied == 0". > > > > (2) One dirty page is not being released on every loop even though > > > > "copied == 0" (at least this problem keeps it from being an infinite > > > > loop by eventually exhausting reserveable space on the disk). > > > > > > Hi everyone, > > > > > > There are actually tow bugs here. First the one that Mitch hit, and a > > > second one that still results in bad file_write results with my > > > debugging hunks (the first two hunks below) in place. > > > > > > My patch fixes Mitch''s bug by checking for copied == 0 after > > > btrfs_copy_from_user and going the correct delalloc accounting. This > > > one looks solved, but you''ll notice the patch is bigger. > > > > > > First, I add some random failures to btrfs_copy_from_user() by failing > > > everyone once and a while. This was much more reliable than trying to > > > use memory pressure than making copy_from_user fail. > > > > > > If copy_from_user fails and we partially update a page, we end up with a > > > page that may go away due to memory pressure. But, btrfs_file_write > > > assumes that only the first and last page may have good data that needs > > > to be read off the disk. > > > > > > This patch ditches that code and puts it into prepare_pages instead. > > > But I''m still having some errors during long stress.sh runs. Ideas are > > > more than welcome, hopefully some other timezones will kick in ideas > > > while I sleep. > > > > At least it doesn''t fix the emerge-problem for me. The behavior is now the same > > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no > > further interaction to get the emerge-process hang with a svn-process > > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the > > spawned svn-process stays and it needs a reboot to get rid of it. > > Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s > looping? Thanks, > > JosefIt behaves the same way here with btrfs-unstable. The output of "cat /proc/$pid/wchan" is 0. // Maria> -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2011/2/28 Maria Wikström <maria@ponstudios.se>:> mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik: >> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote: >> > On Monday 28 February 2011 02:46:05 Chris Mason wrote: >> > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: >> > > > Some clarification on my previous message... >> > > > >> > > > After looking at my ftrace log more closely, I can see where Btrfs is >> > > > trying to release the allocated pages. However, the calculation for >> > > > the number of dirty_pages is equal to 1 when "copied == 0". >> > > > >> > > > So I''m seeing at least two problems: >> > > > (1) It keeps looping when "copied == 0". >> > > > (2) One dirty page is not being released on every loop even though >> > > > "copied == 0" (at least this problem keeps it from being an infinite >> > > > loop by eventually exhausting reserveable space on the disk). >> > > >> > > Hi everyone, >> > > >> > > There are actually tow bugs here. First the one that Mitch hit, and a >> > > second one that still results in bad file_write results with my >> > > debugging hunks (the first two hunks below) in place. >> > > >> > > My patch fixes Mitch''s bug by checking for copied == 0 after >> > > btrfs_copy_from_user and going the correct delalloc accounting. This >> > > one looks solved, but you''ll notice the patch is bigger. >> > > >> > > First, I add some random failures to btrfs_copy_from_user() by failing >> > > everyone once and a while. This was much more reliable than trying to >> > > use memory pressure than making copy_from_user fail. >> > > >> > > If copy_from_user fails and we partially update a page, we end up with a >> > > page that may go away due to memory pressure. But, btrfs_file_write >> > > assumes that only the first and last page may have good data that needs >> > > to be read off the disk. >> > > >> > > This patch ditches that code and puts it into prepare_pages instead. >> > > But I''m still having some errors during long stress.sh runs. Ideas are >> > > more than welcome, hopefully some other timezones will kick in ideas >> > > while I sleep. >> > >> > At least it doesn''t fix the emerge-problem for me. The behavior is now the same >> > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no >> > further interaction to get the emerge-process hang with a svn-process >> > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the >> > spawned svn-process stays and it needs a reboot to get rid of it. >> >> Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s >> looping? Thanks, >> >> Josef > > It behaves the same way here with btrfs-unstable. > The output of "cat /proc/$pid/wchan" is 0. > > // Maria > >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > >I''ve applied the patch at the head of this thread (with the jiffies debugging commented out) and I''m attaching a ftrace using the function_graph tracer when I''m stuck in the loop. I''ve just snipped out a couple of the loops (the full trace file is quite large, and mostly repititious). I''m going to try to modify file.c with some trace_printk debugging to show the values of several of the relevant variables at various stages. I''m going to try to exit the loop after 256 tries with an EFAULT so I can stop the tracing at that point and capture a trace of the entry into the problem (the ftrace ring buffer fills up too fast for me to capture the entry point).
2011/2/28 Mitch Harder <mitch.harder@sabayonlinux.org>:> 2011/2/28 Maria Wikström <maria@ponstudios.se>: >> mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik: >>> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote: >>> > On Monday 28 February 2011 02:46:05 Chris Mason wrote: >>> > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: >>> > > > Some clarification on my previous message... >>> > > > >>> > > > After looking at my ftrace log more closely, I can see where Btrfs is >>> > > > trying to release the allocated pages. However, the calculation for >>> > > > the number of dirty_pages is equal to 1 when "copied == 0". >>> > > > >>> > > > So I''m seeing at least two problems: >>> > > > (1) It keeps looping when "copied == 0". >>> > > > (2) One dirty page is not being released on every loop even though >>> > > > "copied == 0" (at least this problem keeps it from being an infinite >>> > > > loop by eventually exhausting reserveable space on the disk). >>> > > >>> > > Hi everyone, >>> > > >>> > > There are actually tow bugs here. First the one that Mitch hit, and a >>> > > second one that still results in bad file_write results with my >>> > > debugging hunks (the first two hunks below) in place. >>> > > >>> > > My patch fixes Mitch''s bug by checking for copied == 0 after >>> > > btrfs_copy_from_user and going the correct delalloc accounting. This >>> > > one looks solved, but you''ll notice the patch is bigger. >>> > > >>> > > First, I add some random failures to btrfs_copy_from_user() by failing >>> > > everyone once and a while. This was much more reliable than trying to >>> > > use memory pressure than making copy_from_user fail. >>> > > >>> > > If copy_from_user fails and we partially update a page, we end up with a >>> > > page that may go away due to memory pressure. But, btrfs_file_write >>> > > assumes that only the first and last page may have good data that needs >>> > > to be read off the disk. >>> > > >>> > > This patch ditches that code and puts it into prepare_pages instead. >>> > > But I''m still having some errors during long stress.sh runs. Ideas are >>> > > more than welcome, hopefully some other timezones will kick in ideas >>> > > while I sleep. >>> > >>> > At least it doesn''t fix the emerge-problem for me. The behavior is now the same >>> > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no >>> > further interaction to get the emerge-process hang with a svn-process >>> > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the >>> > spawned svn-process stays and it needs a reboot to get rid of it. >>> >>> Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s >>> looping? Thanks, >>> >>> Josef >> >> It behaves the same way here with btrfs-unstable. >> The output of "cat /proc/$pid/wchan" is 0. >> >> // Maria >> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> > > I''ve applied the patch at the head of this thread (with the jiffies > debugging commented out) and I''m attaching a ftrace using the > function_graph tracer when I''m stuck in the loop. I''ve just snipped > out a couple of the loops (the full trace file is quite large, and > mostly repititious). > > I''m going to try to modify file.c with some trace_printk debugging to > show the values of several of the relevant variables at various > stages. > > I''m going to try to exit the loop after 256 tries with an EFAULT so I > can stop the tracing at that point and capture a trace of the entry > into the problem (the ftrace ring buffer fills up too fast for me to > capture the entry point). >As promised, I''m put together a modified file.c with many trace_printk debugging statements to augment the ftrace. The trace is ~128K compressed (about 31,600 lines or 2.6MB uncompressed), so I''m putting it up on my local server instead of attaching. Let me know if it would be more appropriate to send to the list as an attachment. http://dontpanic.dyndns.org/ftrace-btrfs-file-write-debug-v2.gz I preface all my trace_printk comments with "TPK:" to make skipping through the trace easier. The trace contains the trace of about 3 or 4 successful passes through the btrfs_file_aio_write() function to show what a successful trace looks like. The pass through the btrfs_file_aio_write() that breaks begins on line 1088. I let it loop through the while (iov_iter_count(&i) > 0) {} loop for 256 times when copied==0 (otherwise it would loop infinitely). Then exit out and stop the trace. For reference, here''s a diff file for the debugging statements I''ve added to file.c: Let me know if you would like me to re-run this trial with different debugging lines. fs/btrfs/file.c /usr/src/linux/fs/btrfs/file.c --- fs/btrfs/file.c 2011-02-28 10:13:45.000000000 -0600 +++ /usr/src/linux/fs/btrfs/file.c 2011-02-28 13:07:11.000000000 -0600 @@ -53,12 +53,14 @@ int offset = pos & (PAGE_CACHE_SIZE - 1); int total_copied = 0; + /*************************** if ((jiffies % 10) == 0) return 0; if ((jiffies % 25) == 0) { write_bytes /= 2; } + **************************/ while (write_bytes > 0) { size_t count = min_t(size_t, @@ -82,10 +84,13 @@ /* Return to btrfs_file_aio_write to fault page */ if (unlikely(copied == 0)) { + trace_printk("TPK: unlikely copied == 0 in btrfs_copy_from_user (total_copied=%i)\n", + total_copied); break; } if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { + trace_printk("TPK: unlikely copied < PAGE_CACHE_SIZE - offset in btrfs_copy_from_user\n"); offset += copied; } else { pg++; @@ -910,8 +915,13 @@ int will_write; int buffered = 0; int copied = 0; + int copied_problem = 0; + int copied_loop_count = 0; + int stop_ftrace = 0; int dirty_pages = 0; + trace_printk("TPK: Entering btrfs_file_aio_write()\n"); + will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || (file->f_flags & O_DIRECT)); @@ -953,6 +963,7 @@ BTRFS_I(inode)->sequence++; if (unlikely(file->f_flags & O_DIRECT)) { + trace_printk("TPK: transferred to unlikely(file->f_flags \& O_DIRECT)\n"); num_written = generic_file_direct_write(iocb, iov, &nr_segs, pos, ppos, count, ocount); @@ -984,6 +995,8 @@ */ buffered = 1; pos += num_written; + trace_printk("TPK: end unlikely(file->f_flags \& O_DIRECT) with num_written=%i\n", + num_written); } iov_iter_init(&i, iov, nr_segs, count, num_written); @@ -1003,6 +1016,8 @@ last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; while (iov_iter_count(&i) > 0) { + trace_printk("TPK: start section while (iov_iter_count() > 0)\n"); + size_t offset = pos & (PAGE_CACHE_SIZE - 1); size_t write_bytes = min(iov_iter_count(&i), nrptrs * (size_t)PAGE_CACHE_SIZE - @@ -1010,6 +1025,9 @@ size_t num_pages = (write_bytes + offset + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; + trace_printk("TPK: iov_iter_count()=%i || nr_segs=%lu || nrptrs=%i || offset=%lu || write_bytes=%lu || num_pages=%lu\n", + iov_iter_count(&i), nr_segs, nrptrs, offset, write_bytes, num_pages); + WARN_ON(num_pages > nrptrs); memset(pages, 0, sizeof(struct page *) * nrptrs); @@ -1022,6 +1040,19 @@ goto out; } + if (unlikely(copied_problem)) { + copied_loop_count++; + trace_printk("TPK: copied problem(%i)\n", + copied_loop_count); + /* Give up if we''ve already tried 256 times */ + if (copied_loop_count > 256) { + ret = -EFAULT; + stop_ftrace = 1; + trace_printk("TPK: copied loop count exceeded, returning EFAULT....\n"); + goto out; + } + } + ret = btrfs_delalloc_reserve_space(inode, num_pages << PAGE_CACHE_SHIFT); if (ret) @@ -1045,6 +1076,10 @@ PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; + if (copied == 0) { + copied_problem = 1; + } + if (num_pages > dirty_pages) { if (copied > 0) atomic_inc( @@ -1080,11 +1115,19 @@ num_written += copied; cond_resched(); + trace_printk("TPK: end section while (iov_iter_count() > 0)\n"); + trace_printk(" copied=%i || dirty_pages=%i || num_written=%i || ending iov_iter_count=%i\n", + copied, dirty_pages, num_written, iov_iter_count(&i) ); } out: + trace_printk("TPK: Reached: out:\n"); + mutex_unlock(&inode->i_mutex); - if (ret) + if (ret) { err = ret; + trace_printk("TPK: ret,err had value %i when out: was reached (num_written: %i)\n", + ret, num_written); + } kfree(pages); *ppos = pos; @@ -1140,6 +1183,19 @@ } done: current->backing_dev_info = NULL; + if (ret) { + trace_printk("TPK: btrfs_file_aio_write exiting with non-zero ret value (%i)\n", ret); + trace_printk("TPK: Returning num_written (%i) ? num_written (%i) : err (%i) = %i\n", + num_written, num_written, err, num_written ? num_written : err); + } else { + trace_printk("TPK: btrfs_file_aio_write exiting normally with (%i)", + num_written ? num_written : err); + } + + if (unlikely(stop_ftrace)) { + tracing_off(); + } + return num_written ? num_written : err; } -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2011/2/28 Mitch Harder <mitch.harder@sabayonlinux.org>:> 2011/2/28 Mitch Harder <mitch.harder@sabayonlinux.org>: >> 2011/2/28 Maria Wikström <maria@ponstudios.se>: >>> mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik: >>>> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote: >>>> > On Monday 28 February 2011 02:46:05 Chris Mason wrote: >>>> > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: >>>> > > > Some clarification on my previous message... >>>> > > > >>>> > > > After looking at my ftrace log more closely, I can see where Btrfs is >>>> > > > trying to release the allocated pages. However, the calculation for >>>> > > > the number of dirty_pages is equal to 1 when "copied == 0". >>>> > > > >>>> > > > So I''m seeing at least two problems: >>>> > > > (1) It keeps looping when "copied == 0". >>>> > > > (2) One dirty page is not being released on every loop even though >>>> > > > "copied == 0" (at least this problem keeps it from being an infinite >>>> > > > loop by eventually exhausting reserveable space on the disk). >>>> > > >>>> > > Hi everyone, >>>> > > >>>> > > There are actually tow bugs here. First the one that Mitch hit, and a >>>> > > second one that still results in bad file_write results with my >>>> > > debugging hunks (the first two hunks below) in place. >>>> > > >>>> > > My patch fixes Mitch''s bug by checking for copied == 0 after >>>> > > btrfs_copy_from_user and going the correct delalloc accounting. This >>>> > > one looks solved, but you''ll notice the patch is bigger. >>>> > > >>>> > > First, I add some random failures to btrfs_copy_from_user() by failing >>>> > > everyone once and a while. This was much more reliable than trying to >>>> > > use memory pressure than making copy_from_user fail. >>>> > > >>>> > > If copy_from_user fails and we partially update a page, we end up with a >>>> > > page that may go away due to memory pressure. But, btrfs_file_write >>>> > > assumes that only the first and last page may have good data that needs >>>> > > to be read off the disk. >>>> > > >>>> > > This patch ditches that code and puts it into prepare_pages instead. >>>> > > But I''m still having some errors during long stress.sh runs. Ideas are >>>> > > more than welcome, hopefully some other timezones will kick in ideas >>>> > > while I sleep. >>>> > >>>> > At least it doesn''t fix the emerge-problem for me. The behavior is now the same >>>> > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no >>>> > further interaction to get the emerge-process hang with a svn-process >>>> > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the >>>> > spawned svn-process stays and it needs a reboot to get rid of it. >>>> >>>> Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s >>>> looping? Thanks, >>>> >>>> Josef >>> >>> It behaves the same way here with btrfs-unstable. >>> The output of "cat /proc/$pid/wchan" is 0. >>> >>> // Maria >>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>> >>> >> >> I''ve applied the patch at the head of this thread (with the jiffies >> debugging commented out) and I''m attaching a ftrace using the >> function_graph tracer when I''m stuck in the loop. I''ve just snipped >> out a couple of the loops (the full trace file is quite large, and >> mostly repititious). >> >> I''m going to try to modify file.c with some trace_printk debugging to >> show the values of several of the relevant variables at various >> stages. >> >> I''m going to try to exit the loop after 256 tries with an EFAULT so I >> can stop the tracing at that point and capture a trace of the entry >> into the problem (the ftrace ring buffer fills up too fast for me to >> capture the entry point). >> > > As promised, I''m put together a modified file.c with many trace_printk > debugging statements to augment the ftrace. > > The trace is ~128K compressed (about 31,600 lines or 2.6MB > uncompressed), so I''m putting it up on my local server instead of > attaching. Let me know if it would be more appropriate to send to the > list as an attachment. > > http://dontpanic.dyndns.org/ftrace-btrfs-file-write-debug-v2.gz > > I preface all my trace_printk comments with "TPK:" to make skipping > through the trace easier. > > The trace contains the trace of about 3 or 4 successful passes through > the btrfs_file_aio_write() function to show what a successful trace > looks like. > > The pass through the btrfs_file_aio_write() that breaks begins on line 1088. > > I let it loop through the while (iov_iter_count(&i) > 0) {} loop for > 256 times when copied==0 (otherwise it would loop infinitely). Then > exit out and stop the trace. > > For reference, here''s a diff file for the debugging statements I''ve > added to file.c: > > Let me know if you would like me to re-run this trial with different > debugging lines. > > fs/btrfs/file.c /usr/src/linux/fs/btrfs/file.c > --- fs/btrfs/file.c 2011-02-28 10:13:45.000000000 -0600 > +++ /usr/src/linux/fs/btrfs/file.c 2011-02-28 13:07:11.000000000 -0600 > @@ -53,12 +53,14 @@ > int offset = pos & (PAGE_CACHE_SIZE - 1); > int total_copied = 0; > > + /*************************** > if ((jiffies % 10) == 0) > return 0; > > if ((jiffies % 25) == 0) { > write_bytes /= 2; > } > + **************************/ > > while (write_bytes > 0) { > size_t count = min_t(size_t, > @@ -82,10 +84,13 @@ > > /* Return to btrfs_file_aio_write to fault page */ > if (unlikely(copied == 0)) { > + trace_printk("TPK: unlikely copied == 0 in btrfs_copy_from_user > (total_copied=%i)\n", > + total_copied); > break; > } > > if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { > + trace_printk("TPK: unlikely copied < PAGE_CACHE_SIZE - offset in > btrfs_copy_from_user\n"); > offset += copied; > } else { > pg++; > @@ -910,8 +915,13 @@ > int will_write; > int buffered = 0; > int copied = 0; > + int copied_problem = 0; > + int copied_loop_count = 0; > + int stop_ftrace = 0; > int dirty_pages = 0; > > + trace_printk("TPK: Entering btrfs_file_aio_write()\n"); > + > will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || > (file->f_flags & O_DIRECT)); > > @@ -953,6 +963,7 @@ > BTRFS_I(inode)->sequence++; > > if (unlikely(file->f_flags & O_DIRECT)) { > + trace_printk("TPK: transferred to unlikely(file->f_flags \& O_DIRECT)\n"); > num_written = generic_file_direct_write(iocb, iov, &nr_segs, > pos, ppos, count, > ocount); > @@ -984,6 +995,8 @@ > */ > buffered = 1; > pos += num_written; > + trace_printk("TPK: end unlikely(file->f_flags \& O_DIRECT) with > num_written=%i\n", > + num_written); > } > > iov_iter_init(&i, iov, nr_segs, count, num_written); > @@ -1003,6 +1016,8 @@ > last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; > > while (iov_iter_count(&i) > 0) { > + trace_printk("TPK: start section while (iov_iter_count() > 0)\n"); > + > size_t offset = pos & (PAGE_CACHE_SIZE - 1); > size_t write_bytes = min(iov_iter_count(&i), > nrptrs * (size_t)PAGE_CACHE_SIZE - > @@ -1010,6 +1025,9 @@ > size_t num_pages = (write_bytes + offset + > PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; > > + trace_printk("TPK: iov_iter_count()=%i || nr_segs=%lu || nrptrs=%i > || offset=%lu || write_bytes=%lu || num_pages=%lu\n", > + iov_iter_count(&i), nr_segs, nrptrs, offset, write_bytes, num_pages); > + > WARN_ON(num_pages > nrptrs); > memset(pages, 0, sizeof(struct page *) * nrptrs); > > @@ -1022,6 +1040,19 @@ > goto out; > } > > + if (unlikely(copied_problem)) { > + copied_loop_count++; > + trace_printk("TPK: copied problem(%i)\n", > + copied_loop_count); > + /* Give up if we''ve already tried 256 times */ > + if (copied_loop_count > 256) { > + ret = -EFAULT; > + stop_ftrace = 1; > + trace_printk("TPK: copied loop count exceeded, returning EFAULT....\n"); > + goto out; > + } > + } > + > ret = btrfs_delalloc_reserve_space(inode, > num_pages << PAGE_CACHE_SHIFT); > if (ret) > @@ -1045,6 +1076,10 @@ > PAGE_CACHE_SIZE - 1) >> > PAGE_CACHE_SHIFT; > > + if (copied == 0) { > + copied_problem = 1; > + } > + > if (num_pages > dirty_pages) { > if (copied > 0) > atomic_inc( > @@ -1080,11 +1115,19 @@ > num_written += copied; > > cond_resched(); > + trace_printk("TPK: end section while (iov_iter_count() > 0)\n"); > + trace_printk(" copied=%i || dirty_pages=%i || num_written=%i || > ending iov_iter_count=%i\n", > + copied, dirty_pages, num_written, iov_iter_count(&i) ); > } > out: > + trace_printk("TPK: Reached: out:\n"); > + > mutex_unlock(&inode->i_mutex); > - if (ret) > + if (ret) { > err = ret; > + trace_printk("TPK: ret,err had value %i when out: was reached > (num_written: %i)\n", > + ret, num_written); > + } > > kfree(pages); > *ppos = pos; > @@ -1140,6 +1183,19 @@ > } > done: > current->backing_dev_info = NULL; > + if (ret) { > + trace_printk("TPK: btrfs_file_aio_write exiting with non-zero ret > value (%i)\n", ret); > + trace_printk("TPK: Returning num_written (%i) ? num_written (%i) : > err (%i) = %i\n", > + num_written, num_written, err, num_written ? num_written : err); > + } else { > + trace_printk("TPK: btrfs_file_aio_write exiting normally with (%i)", > + num_written ? num_written : err); > + } > + > + if (unlikely(stop_ftrace)) { > + tracing_off(); > + } > + > return num_written ? num_written : err; > } >I''m developing a hypothesis that something is going wrong when Btrfs is trying to lock multiple pages. Page faults are being encountered at the same spot, over and over (BTW, I ran a memcheck to rule that possibility out). If I scan through my traces, it looks like most calls to write are submitted 1 block (4096 bytes) at a time, at the most (also many are < 4096 bytes). The portion that is causing a problem is trying to write 12288 bytes (3k). For some reason, the first 24 bytes are written, and the remainder of the loop is spent on the 12264 that are remaining. But page faults are encountered on each loop, and no more bytes are copied. I modified file.c to cut back on the scope of the attempted write to smaller chunks. The following patch "fixes" my problem, and this build completes without error. I''m not submitting this patch as a solution. It''s clearly papering over a deeper issue. However, I think it gives insight into the problem. I wrote the patch to allow for sequentially smaller blocks if failures continue. But the block cleared once I limited the scope to 4096 bytes. It never needed to try smaller data segments. As hoped, limiting the scope of the write allowed the 12264 bytes to be written in the next three subsequent loops after lowering the scope of the write. It was interesting to note that cutting the scope to 4096 didn''t guarantee that you were limiting the write to one block. There was usually some overlap, and 2 dirty block needed to be written. But I''m still suspicious that there is a mismatch somewhere when trying to lock multiple blocks. Here''s the debugging patch I constructed which actually allowed the build to complete without error. This was applied (for testing purposes) on top of the previous debugging patch. As noted earlier, it never went lower than a 4096 byte window for write_bytes. --- file.c.file-write-patch-v1 2011-02-28 13:06:37.000000000 -0600 +++ file.c 2011-02-28 19:23:21.000000000 -0600 @@ -908,6 +908,7 @@ ssize_t err = 0; size_t count; size_t ocount; + size_t write_bytes = 0; int ret = 0; int nrptrs; unsigned long first_index; @@ -963,7 +964,7 @@ BTRFS_I(inode)->sequence++; if (unlikely(file->f_flags & O_DIRECT)) { - trace_printk("TPK: transferred to unlikely(file->f_flags \& O_DIRECT)\n"); + trace_printk("TPK: transferred to unlikely(file->f_flags & O_DIRECT)\n"); num_written = generic_file_direct_write(iocb, iov, &nr_segs, pos, ppos, count, ocount); @@ -995,7 +996,7 @@ */ buffered = 1; pos += num_written; - trace_printk("TPK: end unlikely(file->f_flags \& O_DIRECT) with num_written=%i\n", + trace_printk("TPK: end unlikely(file->f_flags & O_DIRECT) with num_written=%i\n", num_written); } @@ -1019,9 +1020,20 @@ trace_printk("TPK: start section while (iov_iter_count() > 0)\n"); size_t offset = pos & (PAGE_CACHE_SIZE - 1); - size_t write_bytes = min(iov_iter_count(&i), - nrptrs * (size_t)PAGE_CACHE_SIZE - - offset); + write_bytes = min(iov_iter_count(&i), + nrptrs * (size_t)PAGE_CACHE_SIZE - + offset); + if (unlikely(copied_problem)) { + if (copied_loop_count > 128) { + write_bytes = min(write_bytes, 4096); + } else if (copied_loop_count > 160) { + write_bytes = min(write_bytes, 2048); + } else if (copied_loop_count > 192) { + write_bytes = min(write_bytes, 1024); + } else if (copied_loop_count > 224) { + write_bytes = min(write_bytes, 512); + } + } size_t num_pages = (write_bytes + offset + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Is your system running out of memory or is there any other thread like flush-btrfs competing for the same page? I can only see one process in your ftrace log. You may need to trace all btrfs.ko function calls instead of a single process. Thanks! -----Original Message----- From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-owner@vger.kernel.org] On Behalf Of Mitch Harder Sent: Tuesday, March 01, 2011 4:20 AM To: Maria Wikström Cc: Josef Bacik; Johannes Hirte; Chris Mason; Zhong, Xin; linux-btrfs@vger.kernel.org Subject: Re: [PATCH] btrfs file write debugging patch 2011/2/28 Mitch Harder <mitch.harder@sabayonlinux.org>:> 2011/2/28 Maria Wikström <maria@ponstudios.se>: >> mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik: >>> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote: >>> > On Monday 28 February 2011 02:46:05 Chris Mason wrote: >>> > > Excerpts from Mitch Harder''s message of 2011-02-25 13:43:37 -0500: >>> > > > Some clarification on my previous message... >>> > > > >>> > > > After looking at my ftrace log more closely, I can see where Btrfs is >>> > > > trying to release the allocated pages. However, the calculation for >>> > > > the number of dirty_pages is equal to 1 when "copied == 0". >>> > > > >>> > > > So I''m seeing at least two problems: >>> > > > (1) It keeps looping when "copied == 0". >>> > > > (2) One dirty page is not being released on every loop even though >>> > > > "copied == 0" (at least this problem keeps it from being an infinite >>> > > > loop by eventually exhausting reserveable space on the disk). >>> > > >>> > > Hi everyone, >>> > > >>> > > There are actually tow bugs here. First the one that Mitch hit, and a >>> > > second one that still results in bad file_write results with my >>> > > debugging hunks (the first two hunks below) in place. >>> > > >>> > > My patch fixes Mitch''s bug by checking for copied == 0 after >>> > > btrfs_copy_from_user and going the correct delalloc accounting. This >>> > > one looks solved, but you''ll notice the patch is bigger. >>> > > >>> > > First, I add some random failures to btrfs_copy_from_user() by failing >>> > > everyone once and a while. This was much more reliable than trying to >>> > > use memory pressure than making copy_from_user fail. >>> > > >>> > > If copy_from_user fails and we partially update a page, we end up with a >>> > > page that may go away due to memory pressure. But, btrfs_file_write >>> > > assumes that only the first and last page may have good data that needs >>> > > to be read off the disk. >>> > > >>> > > This patch ditches that code and puts it into prepare_pages instead. >>> > > But I''m still having some errors during long stress.sh runs. Ideas are >>> > > more than welcome, hopefully some other timezones will kick in ideas >>> > > while I sleep. >>> > >>> > At least it doesn''t fix the emerge-problem for me. The behavior is now the same >>> > as with 2.6.38-rc3. It needs a ''emerge --oneshot dev-libs/libgcrypt'' with no >>> > further interaction to get the emerge-process hang with a svn-process >>> > consuming 100% CPU. I can cancel the emerge-process with ctrl-c but the >>> > spawned svn-process stays and it needs a reboot to get rid of it. >>> >>> Can you cat /proc/$pid/wchan a few times so we can get an idea of where it''s >>> looping? Thanks, >>> >>> Josef >> >> It behaves the same way here with btrfs-unstable. >> The output of "cat /proc/$pid/wchan" is 0. >> >> // Maria >> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> > > I''ve applied the patch at the head of this thread (with the jiffies > debugging commented out) and I''m attaching a ftrace using the > function_graph tracer when I''m stuck in the loop. I''ve just snipped > out a couple of the loops (the full trace file is quite large, and > mostly repititious). > > I''m going to try to modify file.c with some trace_printk debugging to > show the values of several of the relevant variables at various > stages. > > I''m going to try to exit the loop after 256 tries with an EFAULT so I > can stop the tracing at that point and capture a trace of the entry > into the problem (the ftrace ring buffer fills up too fast for me to > capture the entry point). >As promised, I''m put together a modified file.c with many trace_printk debugging statements to augment the ftrace. The trace is ~128K compressed (about 31,600 lines or 2.6MB uncompressed), so I''m putting it up on my local server instead of attaching. Let me know if it would be more appropriate to send to the list as an attachment. http://dontpanic.dyndns.org/ftrace-btrfs-file-write-debug-v2.gz I preface all my trace_printk comments with "TPK:" to make skipping through the trace easier. The trace contains the trace of about 3 or 4 successful passes through the btrfs_file_aio_write() function to show what a successful trace looks like. The pass through the btrfs_file_aio_write() that breaks begins on line 1088. I let it loop through the while (iov_iter_count(&i) > 0) {} loop for 256 times when copied==0 (otherwise it would loop infinitely). Then exit out and stop the trace. For reference, here''s a diff file for the debugging statements I''ve added to file.c: Let me know if you would like me to re-run this trial with different debugging lines. fs/btrfs/file.c /usr/src/linux/fs/btrfs/file.c --- fs/btrfs/file.c 2011-02-28 10:13:45.000000000 -0600 +++ /usr/src/linux/fs/btrfs/file.c 2011-02-28 13:07:11.000000000 -0600 @@ -53,12 +53,14 @@ int offset = pos & (PAGE_CACHE_SIZE - 1); int total_copied = 0; + /*************************** if ((jiffies % 10) == 0) return 0; if ((jiffies % 25) == 0) { write_bytes /= 2; } + **************************/ while (write_bytes > 0) { size_t count = min_t(size_t, @@ -82,10 +84,13 @@ /* Return to btrfs_file_aio_write to fault page */ if (unlikely(copied == 0)) { + trace_printk("TPK: unlikely copied == 0 in btrfs_copy_from_user (total_copied=%i)\n", + total_copied); break; } if (unlikely(copied < PAGE_CACHE_SIZE - offset)) { + trace_printk("TPK: unlikely copied < PAGE_CACHE_SIZE - offset in btrfs_copy_from_user\n"); offset += copied; } else { pg++; @@ -910,8 +915,13 @@ int will_write; int buffered = 0; int copied = 0; + int copied_problem = 0; + int copied_loop_count = 0; + int stop_ftrace = 0; int dirty_pages = 0; + trace_printk("TPK: Entering btrfs_file_aio_write()\n"); + will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) || (file->f_flags & O_DIRECT)); @@ -953,6 +963,7 @@ BTRFS_I(inode)->sequence++; if (unlikely(file->f_flags & O_DIRECT)) { + trace_printk("TPK: transferred to unlikely(file->f_flags \& O_DIRECT)\n"); num_written = generic_file_direct_write(iocb, iov, &nr_segs, pos, ppos, count, ocount); @@ -984,6 +995,8 @@ */ buffered = 1; pos += num_written; + trace_printk("TPK: end unlikely(file->f_flags \& O_DIRECT) with num_written=%i\n", + num_written); } iov_iter_init(&i, iov, nr_segs, count, num_written); @@ -1003,6 +1016,8 @@ last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT; while (iov_iter_count(&i) > 0) { + trace_printk("TPK: start section while (iov_iter_count() > 0)\n"); + size_t offset = pos & (PAGE_CACHE_SIZE - 1); size_t write_bytes = min(iov_iter_count(&i), nrptrs * (size_t)PAGE_CACHE_SIZE - @@ -1010,6 +1025,9 @@ size_t num_pages = (write_bytes + offset + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; + trace_printk("TPK: iov_iter_count()=%i || nr_segs=%lu || nrptrs=%i || offset=%lu || write_bytes=%lu || num_pages=%lu\n", + iov_iter_count(&i), nr_segs, nrptrs, offset, write_bytes, num_pages); + WARN_ON(num_pages > nrptrs); memset(pages, 0, sizeof(struct page *) * nrptrs); @@ -1022,6 +1040,19 @@ goto out; } + if (unlikely(copied_problem)) { + copied_loop_count++; + trace_printk("TPK: copied problem(%i)\n", + copied_loop_count); + /* Give up if we''ve already tried 256 times */ + if (copied_loop_count > 256) { + ret = -EFAULT; + stop_ftrace = 1; + trace_printk("TPK: copied loop count exceeded, returning EFAULT....\n"); + goto out; + } + } + ret = btrfs_delalloc_reserve_space(inode, num_pages << PAGE_CACHE_SHIFT); if (ret) @@ -1045,6 +1076,10 @@ PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT; + if (copied == 0) { + copied_problem = 1; + } + if (num_pages > dirty_pages) { if (copied > 0) atomic_inc( @@ -1080,11 +1115,19 @@ num_written += copied; cond_resched(); + trace_printk("TPK: end section while (iov_iter_count() > 0)\n"); + trace_printk(" copied=%i || dirty_pages=%i || num_written=%i || ending iov_iter_count=%i\n", + copied, dirty_pages, num_written, iov_iter_count(&i) ); } out: + trace_printk("TPK: Reached: out:\n"); + mutex_unlock(&inode->i_mutex); - if (ret) + if (ret) { err = ret; + trace_printk("TPK: ret,err had value %i when out: was reached (num_written: %i)\n", + ret, num_written); + } kfree(pages); *ppos = pos; @@ -1140,6 +1183,19 @@ } done: current->backing_dev_info = NULL; + if (ret) { + trace_printk("TPK: btrfs_file_aio_write exiting with non-zero ret value (%i)\n", ret); + trace_printk("TPK: Returning num_written (%i) ? num_written (%i) : err (%i) = %i\n", + num_written, num_written, err, num_written ? num_written : err); + } else { + trace_printk("TPK: btrfs_file_aio_write exiting normally with (%i)", + num_written ? num_written : err); + } + + if (unlikely(stop_ftrace)) { + tracing_off(); + } + return num_written ? num_written : err; } -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Mitch,
I suspect there''s a lock contention between flush-btrfs
(lock_dellalloc_pages) and btrfs_file_aio_write. However I can not recreate it
locally. Could you please try below patch? Thanks!
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 65338a1..b9d0929 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1007,17 +1007,16 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb,
goto out;
}
- ret = btrfs_delalloc_reserve_space(inode,
- num_pages << PAGE_CACHE_SHIFT);
- if (ret)
- goto out;
-
ret = prepare_pages(root, file, pages, num_pages,
pos, first_index, last_index,
write_bytes);
- if (ret) {
- btrfs_delalloc_release_space(inode,
+ if (ret)
+ goto out;
+
+ ret = btrfs_delalloc_reserve_space(inode,
num_pages << PAGE_CACHE_SHIFT);
+ if (ret) {
+ btrfs_drop_pages(pages, num_pages);
goto out;
}
-----Original Message-----
From: linux-btrfs-owner@vger.kernel.org
[mailto:linux-btrfs-owner@vger.kernel.org] On Behalf Of Zhong, Xin
Sent: Tuesday, March 01, 2011 6:15 PM
To: Mitch Harder; Maria Wikström
Cc: Josef Bacik; Johannes Hirte; Chris Mason; linux-btrfs@vger.kernel.org
Subject: RE: [PATCH] btrfs file write debugging patch
Is your system running out of memory or is there any other thread like
flush-btrfs competing for the same page?
I can only see one process in your ftrace log. You may need to trace all
btrfs.ko function calls instead of a single process. Thanks!
-----Original Message-----
From: linux-btrfs-owner@vger.kernel.org
[mailto:linux-btrfs-owner@vger.kernel.org] On Behalf Of Mitch Harder
Sent: Tuesday, March 01, 2011 4:20 AM
To: Maria Wikström
Cc: Josef Bacik; Johannes Hirte; Chris Mason; Zhong, Xin;
linux-btrfs@vger.kernel.org
Subject: Re: [PATCH] btrfs file write debugging patch
2011/2/28 Mitch Harder
<mitch.harder@sabayonlinux.org>:> 2011/2/28 Maria Wikström <maria@ponstudios.se>:
>> mån 2011-02-28 klockan 11:10 -0500 skrev Josef Bacik:
>>> On Mon, Feb 28, 2011 at 11:13:59AM +0100, Johannes Hirte wrote:
>>> > On Monday 28 February 2011 02:46:05 Chris Mason wrote:
>>> > > Excerpts from Mitch Harder''s message of
2011-02-25 13:43:37 -0500:
>>> > > > Some clarification on my previous message...
>>> > > >
>>> > > > After looking at my ftrace log more closely, I can
see where Btrfs is
>>> > > > trying to release the allocated pages. However, the
calculation for
>>> > > > the number of dirty_pages is equal to 1 when
"copied == 0".
>>> > > >
>>> > > > So I''m seeing at least two problems:
>>> > > > (1) It keeps looping when "copied == 0".
>>> > > > (2) One dirty page is not being released on every
loop even though
>>> > > > "copied == 0" (at least this problem keeps
it from being an infinite
>>> > > > loop by eventually exhausting reserveable space on
the disk).
>>> > >
>>> > > Hi everyone,
>>> > >
>>> > > There are actually tow bugs here. First the one that
Mitch hit, and a
>>> > > second one that still results in bad file_write results
with my
>>> > > debugging hunks (the first two hunks below) in place.
>>> > >
>>> > > My patch fixes Mitch''s bug by checking for
copied == 0 after
>>> > > btrfs_copy_from_user and going the correct delalloc
accounting. This
>>> > > one looks solved, but you''ll notice the patch is
bigger.
>>> > >
>>> > > First, I add some random failures to
btrfs_copy_from_user() by failing
>>> > > everyone once and a while. This was much more reliable
than trying to
>>> > > use memory pressure than making copy_from_user fail.
>>> > >
>>> > > If copy_from_user fails and we partially update a page,
we end up with a
>>> > > page that may go away due to memory pressure. But,
btrfs_file_write
>>> > > assumes that only the first and last page may have good
data that needs
>>> > > to be read off the disk.
>>> > >
>>> > > This patch ditches that code and puts it into
prepare_pages instead.
>>> > > But I''m still having some errors during long
stress.sh runs. Ideas are
>>> > > more than welcome, hopefully some other timezones will
kick in ideas
>>> > > while I sleep.
>>> >
>>> > At least it doesn''t fix the emerge-problem for me.
The behavior is now the same
>>> > as with 2.6.38-rc3. It needs a ''emerge --oneshot
dev-libs/libgcrypt'' with no
>>> > further interaction to get the emerge-process hang with a
svn-process
>>> > consuming 100% CPU. I can cancel the emerge-process with
ctrl-c but the
>>> > spawned svn-process stays and it needs a reboot to get rid of
it.
>>>
>>> Can you cat /proc/$pid/wchan a few times so we can get an idea of
where it''s
>>> looping? Thanks,
>>>
>>> Josef
>>
>> It behaves the same way here with btrfs-unstable.
>> The output of "cat /proc/$pid/wchan" is 0.
>>
>> // Maria
>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>>
>
> I''ve applied the patch at the head of this thread (with the
jiffies
> debugging commented out) and I''m attaching a ftrace using the
> function_graph tracer when I''m stuck in the loop. I''ve
just snipped
> out a couple of the loops (the full trace file is quite large, and
> mostly repititious).
>
> I''m going to try to modify file.c with some trace_printk debugging
to
> show the values of several of the relevant variables at various
> stages.
>
> I''m going to try to exit the loop after 256 tries with an EFAULT
so I
> can stop the tracing at that point and capture a trace of the entry
> into the problem (the ftrace ring buffer fills up too fast for me to
> capture the entry point).
>
As promised, I''m put together a modified file.c with many trace_printk
debugging statements to augment the ftrace.
The trace is ~128K compressed (about 31,600 lines or 2.6MB
uncompressed), so I''m putting it up on my local server instead of
attaching. Let me know if it would be more appropriate to send to the
list as an attachment.
http://dontpanic.dyndns.org/ftrace-btrfs-file-write-debug-v2.gz
I preface all my trace_printk comments with "TPK:" to make skipping
through the trace easier.
The trace contains the trace of about 3 or 4 successful passes through
the btrfs_file_aio_write() function to show what a successful trace
looks like.
The pass through the btrfs_file_aio_write() that breaks begins on line 1088.
I let it loop through the while (iov_iter_count(&i) > 0) {} loop for
256 times when copied==0 (otherwise it would loop infinitely). Then
exit out and stop the trace.
For reference, here''s a diff file for the debugging statements
I''ve
added to file.c:
Let me know if you would like me to re-run this trial with different
debugging lines.
fs/btrfs/file.c /usr/src/linux/fs/btrfs/file.c
--- fs/btrfs/file.c 2011-02-28 10:13:45.000000000 -0600
+++ /usr/src/linux/fs/btrfs/file.c 2011-02-28 13:07:11.000000000 -0600
@@ -53,12 +53,14 @@
int offset = pos & (PAGE_CACHE_SIZE - 1);
int total_copied = 0;
+ /***************************
if ((jiffies % 10) == 0)
return 0;
if ((jiffies % 25) == 0) {
write_bytes /= 2;
}
+ **************************/
while (write_bytes > 0) {
size_t count = min_t(size_t,
@@ -82,10 +84,13 @@
/* Return to btrfs_file_aio_write to fault page */
if (unlikely(copied == 0)) {
+ trace_printk("TPK: unlikely copied == 0 in btrfs_copy_from_user
(total_copied=%i)\n",
+ total_copied);
break;
}
if (unlikely(copied < PAGE_CACHE_SIZE - offset)) {
+ trace_printk("TPK: unlikely copied < PAGE_CACHE_SIZE - offset in
btrfs_copy_from_user\n");
offset += copied;
} else {
pg++;
@@ -910,8 +915,13 @@
int will_write;
int buffered = 0;
int copied = 0;
+ int copied_problem = 0;
+ int copied_loop_count = 0;
+ int stop_ftrace = 0;
int dirty_pages = 0;
+ trace_printk("TPK: Entering btrfs_file_aio_write()\n");
+
will_write = ((file->f_flags & O_DSYNC) || IS_SYNC(inode) ||
(file->f_flags & O_DIRECT));
@@ -953,6 +963,7 @@
BTRFS_I(inode)->sequence++;
if (unlikely(file->f_flags & O_DIRECT)) {
+ trace_printk("TPK: transferred to unlikely(file->f_flags \&
O_DIRECT)\n");
num_written = generic_file_direct_write(iocb, iov, &nr_segs,
pos, ppos, count,
ocount);
@@ -984,6 +995,8 @@
*/
buffered = 1;
pos += num_written;
+ trace_printk("TPK: end unlikely(file->f_flags \& O_DIRECT) with
num_written=%i\n",
+ num_written);
}
iov_iter_init(&i, iov, nr_segs, count, num_written);
@@ -1003,6 +1016,8 @@
last_index = (pos + iov_iter_count(&i)) >> PAGE_CACHE_SHIFT;
while (iov_iter_count(&i) > 0) {
+ trace_printk("TPK: start section while (iov_iter_count() >
0)\n");
+
size_t offset = pos & (PAGE_CACHE_SIZE - 1);
size_t write_bytes = min(iov_iter_count(&i),
nrptrs * (size_t)PAGE_CACHE_SIZE -
@@ -1010,6 +1025,9 @@
size_t num_pages = (write_bytes + offset +
PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+ trace_printk("TPK: iov_iter_count()=%i || nr_segs=%lu || nrptrs=%i
|| offset=%lu || write_bytes=%lu || num_pages=%lu\n",
+ iov_iter_count(&i), nr_segs, nrptrs, offset, write_bytes, num_pages);
+
WARN_ON(num_pages > nrptrs);
memset(pages, 0, sizeof(struct page *) * nrptrs);
@@ -1022,6 +1040,19 @@
goto out;
}
+ if (unlikely(copied_problem)) {
+ copied_loop_count++;
+ trace_printk("TPK: copied problem(%i)\n",
+ copied_loop_count);
+ /* Give up if we''ve already tried 256 times */
+ if (copied_loop_count > 256) {
+ ret = -EFAULT;
+ stop_ftrace = 1;
+ trace_printk("TPK: copied loop count exceeded, returning
EFAULT....\n");
+ goto out;
+ }
+ }
+
ret = btrfs_delalloc_reserve_space(inode,
num_pages << PAGE_CACHE_SHIFT);
if (ret)
@@ -1045,6 +1076,10 @@
PAGE_CACHE_SIZE - 1) >>
PAGE_CACHE_SHIFT;
+ if (copied == 0) {
+ copied_problem = 1;
+ }
+
if (num_pages > dirty_pages) {
if (copied > 0)
atomic_inc(
@@ -1080,11 +1115,19 @@
num_written += copied;
cond_resched();
+ trace_printk("TPK: end section while (iov_iter_count() > 0)\n");
+ trace_printk(" copied=%i || dirty_pages=%i || num_written=%i ||
ending iov_iter_count=%i\n",
+ copied, dirty_pages, num_written, iov_iter_count(&i) );
}
out:
+ trace_printk("TPK: Reached: out:\n");
+
mutex_unlock(&inode->i_mutex);
- if (ret)
+ if (ret) {
err = ret;
+ trace_printk("TPK: ret,err had value %i when out: was reached
(num_written: %i)\n",
+ ret, num_written);
+ }
kfree(pages);
*ppos = pos;
@@ -1140,6 +1183,19 @@
}
done:
current->backing_dev_info = NULL;
+ if (ret) {
+ trace_printk("TPK: btrfs_file_aio_write exiting with non-zero ret
value (%i)\n", ret);
+ trace_printk("TPK: Returning num_written (%i) ? num_written (%i) :
err (%i) = %i\n",
+ num_written, num_written, err, num_written ? num_written : err);
+ } else {
+ trace_printk("TPK: btrfs_file_aio_write exiting normally with
(%i)",
+ num_written ? num_written : err);
+ }
+
+ if (unlikely(stop_ftrace)) {
+ tracing_off();
+ }
+
return num_written ? num_written : err;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 1, 2011 at 4:14 AM, Zhong, Xin <xin.zhong@intel.com> wrote:> Is your system running out of memory or is there any other thread like flush-btrfs competing for the same page? >There''s no sign of memory pressure. Although I only have 1 GB in this box, I''m still show ~1/2 GB RAM free during this build. There''s no swap space allocated, and nothing in dmesg that indicates there''s a transient spike of RAM pressure.> I can only see one process in your ftrace log. You may need to trace all btrfs.ko function calls instead of a single process. Thanks! >That ftrace.log was run with ftrace defaults for a function trace. It should collect calls from the whole system. For the sake of consistency, I am intentionally trying to insure that very few other things are going on at the same time as this build. And I''m building with "-j1" so things will happen the same way each time. Also, I supplied just the tail end of the trace log. The full log shows a few of the other build processes leading up to the problem, but the ftrace ring buffer fills up surprisingly fast. Even with a 50MB ring buffer for ftrace, I usually collect less than 1 second of information when something busy like a build is going on. Let me know if you''d like to see the full log. It''s bigger, but I can find someplace to put it. But I''m pretty sure that wmldbcreate is the only thing that is going on when the breakage occurs. Otherwise I wouldn''t get such consistent breakage in the same spot every time. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 1, 2011 at 5:56 AM, Zhong, Xin <xin.zhong@intel.com> wrote:> Hi Mitch, > > I suspect there''s a lock contention between flush-btrfs (lock_dellalloc_pages) and btrfs_file_aio_write. However I can not recreate it locally. Could you please try below patch? Thanks! > > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 65338a1..b9d0929 100644 > --- a/fs/btrfs/file.c > +++ b/fs/btrfs/file.c > @@ -1007,17 +1007,16 @@ static ssize_t btrfs_file_aio_write(struct kiocb *iocb, > goto out; > } > > - ret = btrfs_delalloc_reserve_space(inode, > - num_pages << PAGE_CACHE_SHIFT); > - if (ret) > - goto out; > - > ret = prepare_pages(root, file, pages, num_pages, > pos, first_index, last_index, > write_bytes); > - if (ret) { > - btrfs_delalloc_release_space(inode, > + if (ret) > + goto out; > + > + ret = btrfs_delalloc_reserve_space(inode, > num_pages << PAGE_CACHE_SHIFT); > + if (ret) { > + btrfs_drop_pages(pages, num_pages); > goto out; > } > >Thanks. I''ve tested this patch, but the build is still failing at the same point as before. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 28, 2011 at 02:20:22PM -0600, Mitch Harder wrote:> As promised, I''m put together a modified file.c with many trace_printk > debugging statements to augment the ftrace. > *snip*Just my few cents. I''ve applied the patch from Chris Mason (Sun, 27 Feb 2011 20:46:05 -0500) and this one from Mitch (Mon, 28 Feb 2011 14:20:22 -0600) on top of vanilla 2.6.38-rc6 and it seems that it resolves my issues with hanging `svn info'' during libgcrypt emerge. Piotr Szymaniak. -- - (...) Nie wyobrazam sobie, co ta gora miesa moglaby ci dac, czego ja nie moglbym ofiarowac. Oczywiscie poza piecdziesiecioma funtami rozrosnietych miesni. - Moze mnie wlasnie pociagaja rozrosniete miesnie. (...) W koncu wielu mezczyzn pociaga rozrosnieta tkanka tluszczowa piersi. -- Graham Masterton, "The Wells of Hell"