Filipe Brandenburger
2013-Feb-12 08:37 UTC
[RFC][PATCH 0/2] btrfs-progs: introduce kernelsrc directory to update it
Hello, This is a proposal on a way to keep btrfs-progs in sync with kernel code. https://btrfs.wiki.kernel.org/index.php/Cleanup_ideas#Use_the_kernel_code_in_user_mode For now, the patch introduces a method to import and update the kernel source files into the btrfs-progs tree. Further work will be needed to update the kernel source code so to make it generic (i.e. add #define''s and #ifdef''s) so that it can be compiled in both kernel and user space. The Python script is just a draft, it''s already functional but error handling (and reporting) could definitely be improved, the code could be cleaned up, it could cache the kernel source directory and allow a --force parameter to allow moving to an tag that''s not a descendant, but it''s already fully functional and it''s probably a good start... To see it in action, try this after applying the commits: $ kernelsrc/update_kernelsrc.py /path/to/linux v3.6 $ git status $ git commit -m "btrfs-progs: initial checkout of kernelsrc v3.6" $ vim kernelsrc/update_kernelsrc.filelist (add/delete a line) $ kernelsrc/update_kernelsrc.py ../linux-btrfs/ v3.7 $ git status $ git commit -m "btrfs-progs: upgrade kernelsrc from v3.6 to v3.7" Let me know what you think and if you think it''s worth to continue on this path. Cheers, Filipe Filipe Brandenburger (2): btrfs-progs: anchor gitignore patterns on root btrfs-progs: introduce kernelsrc directory and script to update it .gitignore | 32 +++++--- kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++ kernelsrc/update_kernelsrc.filelist | 23 ++++++ kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++ 4 files changed, 308 insertions(+), 13 deletions(-) create mode 100644 kernelsrc/README.kernelsrc create mode 100644 kernelsrc/update_kernelsrc.filelist create mode 100755 kernelsrc/update_kernelsrc.py -- 1.8.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Filipe Brandenburger
2013-Feb-12 08:37 UTC
[PATCH 1/2] btrfs-progs: anchor gitignore patterns on the root of the tree
This is needed in order to allow checking out fs/btrfs/, otherwise the ignore rule for the btrfs binary would conflict with it. Signed-off-by: Filipe Brandenburger <filbranden@google.com> --- .gitignore | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/.gitignore b/.gitignore index 0e560d5..3d360b6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,15 +1,21 @@ *.o .*.o.d -version.h -man/*.gz -btrfs -btrfs-debug-tree -btrfs-map-logical -btrfs-show -btrfs-vol -btrfsck -btrfsctl -find-root -mkfs.btrfs -repair -restore +/version.h +/man/*.gz +/btrfs +/btrfsck +/btrfs-convert +/btrfsctl +/btrfs-debug-tree +/btrfs-find-root +/btrfs-image +/btrfs-map-logical +/btrfs-restore +/btrfs-show +/btrfstune +/btrfs-vol +/btrfs-zero-log +/find-root +/mkfs.btrfs +/repair +/restore -- 1.8.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Filipe Brandenburger
2013-Feb-12 08:37 UTC
[PATCH 2/2] btrfs-progs: introduce kernelsrc directory and script to update it
The kernelsrc directory will contain a copy of the Btrfs-related files from the
Linux kernel source tree (mainly the files in fs/btrfs/ directory.)
This commit introduces an initial filelist (generated by comparing the list of
files in both directories) and a script to fetch the files from a local clone of
the kernel git directory into the kernelsrc directory. When updating the
directory, a git tag needs to be specified, which is recorded in a local file,
in a way that it is possible to track down to which kernel release the kernelsrc
files originally belong.
A README file was also added to explain the idea behind it and to document the
procedure to update the kernelsrc directory.
Signed-off-by: Filipe Brandenburger <filbranden@google.com>
---
kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++
kernelsrc/update_kernelsrc.filelist | 23 ++++++
kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++
3 files changed, 289 insertions(+)
create mode 100644 kernelsrc/README.kernelsrc
create mode 100644 kernelsrc/update_kernelsrc.filelist
create mode 100755 kernelsrc/update_kernelsrc.py
diff --git a/kernelsrc/README.kernelsrc b/kernelsrc/README.kernelsrc
new file mode 100644
index 0000000..0aed4bb
--- /dev/null
+++ b/kernelsrc/README.kernelsrc
@@ -0,0 +1,108 @@
+
+The kernelsrc directory tree
+===========================+
+The `kernelsrc'' directory in the btrfs-progs tree contains a copy of
some of the
+files in the Linux kernel tree, mainly the files in fs/btrfs/, that are shared
+between the implementation of the filesystem kernel module and some of the
+command-line utilities.
+
+These files should never be edited on the btrfs-progs tree. Changes to these
+files should be committed and pushed to the Linux kernel git tree (optionally
+using #define''s and #ifdef''s to make code compatible with
both kernel and user
+spaces) and then it should be updated here from the kernel tree.
+
+In order to update the `kernelsrc'' directory, you need to have a
checkout of
+both the btrfs-progs git tree and the Linux kernel git tree.
+
+Additionally to the kernel source files, the `kernelsrc'' directory
tree also
+contains the scripts and control files used to update this same tree.
+
+
+Updating the kernelsrc tree with update_kernelsrc.py
+===================================================+
+In order to update the `kernelsrc'' tree to include the source code
files from a
+newer version of the Linux kernel tree, these are the steps that should be
+followed:
+
+ 1. Make sure that you are in the root of the btrfs-progs git tree and that
+ the tree is clean (i.e., run `git status'' and check its output.)
+
+ 2. Make sure you have a clone of the Linux kernel git tree. Let''s
assume
+ for this example that your kernel git tree is in directory ../linux/
+
+ 3. Make sure you have know to which tag of the Linux kernel you want to
+ upgrade the `kernelsrc'' tree to (e.g. v3.7 or v3.8-rc7)
+
+ 4. (Optional:) You may change the list of files that will be copied from
+ the Linux kernel tree. In order to do that, edit the file
+ `kernelsrc/update_kernelsrc.filelist'' and either add or remove
entries
+ from that file.
+
+ 5. Run the following command from the root of the btrfs-progs git tree:
+
+ $ kernelsrc/update_kernelsrc.py ../linux/ v3.7
+
+ (assuming Linux kernel git tree at ../linux/ and upgrading to tag v3.7)
+
+ 6. Check whether the update matches your expectations. In particular, the
+ commands `git status'' and `git diff --cached'' may be
helpful.
+
+ 7. Build btrfs-progs (using `make'') and test them to make sure
they are
+ still compatible with the new kernel source tree. Any needed adjustments
+ may be done by editing files *OUTSIDE* of the `kernelsrc'' tree. Once
+ everything seems to be in place and working correctly, you may add those
+ files to the same git commit using the `git add'' command.
+
+ 8. Commit the change, using the `git commit'' command. You may use
the
+ suggested commit command from the script output, with a standard commit
+ message. The editor will open anyways, so you can add more details
+ (particularly about files outside the `kernelsrc'' tree that needed
+ changes.)
+
+
+Migration to kernelsrc
+=====================+
+Using the `kernelsrc'' tree allows for a gradual migration from the
current
+situation, where copies of the original source files from the kernel tree were
+made in btrfs-progs, adjustments were made to allow them to compile in user
+space, and they diverged further over time.
+
+First, it allows checking out the files from the kernel tree and adding them to
+the btrfs-progs without actually using them initially. Then, it allows
gradually
+migrating the files one by one. One way to do it would be replacing the file in
+the btrfs-progs root directory with a symbolic link to the same file in the
+`kernelsrc'' tree. Another way would be integrating the build of the
`kernelsrc''
+files into the Makefile (assuming all of them compile in userspace without
+errors) so that they get built and can be linked from there. The integration
may
+be switched from the former to the latter at any point in a commit that
+basically only touches the Makefiles and symlinks.
+
+
+Alternatives to using a script to update kernelsrc
+=================================================+
+When coming up with the idea for `update_kernelsrc.py'', my first idea
was to use
+`git submodule'' instead. It seems to perfectly capture the idea of
having a
+repository with source code files that are used by a project in another
+repository. So, in theory, it would be possible to have the Linux kernel git
+tree as a submodule of the btrfs-progs source tree (it could even be placed
+under a `kernelsrc'' subdirectory.) Git would then track the revision
of the
+Linux git tree, and it would be possible to upgrade the tree by doing a
checkout
+on the Linux tree and then committing on the btrfs-progs tree, which would
+simply record the git revision where the Linux tree was pointing to at the time
+of the commit.
+
+The main problem with this approach is the size of the Linux git tree. A clone
+of the btrfs-progs tree takes about 2 or 3 MB of diskspace, while the Linux
tree
+takes about 1 GB. I tried to look at options of doing a `sparse
checkout'' in
+git, but it doesn''t seem to be possible to do it for a submodule...
Even if it
+was, I''m not fully convinced that it would reduce the diskspace
requirements, as
+it would probably still need to fetch the objects into the .git tree anyways.
+
+So, the idea of a script that would update the tree by copying the files and
+keeping the revision in control files seemed like a close enough solution that
+would implement a similar concept without the diskspace problems.
+
diff --git a/kernelsrc/update_kernelsrc.filelist
b/kernelsrc/update_kernelsrc.filelist
new file mode 100644
index 0000000..50cdf95
--- /dev/null
+++ b/kernelsrc/update_kernelsrc.filelist
@@ -0,0 +1,23 @@
+fs/btrfs/ctree.c
+fs/btrfs/ctree.h
+fs/btrfs/dir-item.c
+fs/btrfs/disk-io.c
+fs/btrfs/disk-io.h
+fs/btrfs/extent_io.c
+fs/btrfs/extent_io.h
+fs/btrfs/extent-tree.c
+fs/btrfs/file-item.c
+fs/btrfs/hash.h
+fs/btrfs/inode-item.c
+fs/btrfs/inode-map.c
+fs/btrfs/ioctl.h
+fs/btrfs/print-tree.c
+fs/btrfs/print-tree.h
+fs/btrfs/qgroup.c
+fs/btrfs/root-tree.c
+fs/btrfs/send.h
+fs/btrfs/transaction.h
+fs/btrfs/version.h
+fs/btrfs/volumes.c
+fs/btrfs/volumes.h
+include/linux/list.h
diff --git a/kernelsrc/update_kernelsrc.py b/kernelsrc/update_kernelsrc.py
new file mode 100755
index 0000000..0608671
--- /dev/null
+++ b/kernelsrc/update_kernelsrc.py
@@ -0,0 +1,158 @@
+#! /usr/bin/python
+
+import os
+import subprocess
+import sys
+
+SCRIPT_NAME = ''kernelsrc/update_kernelsrc.py''
+KERNELSRC_GIT_DIR = None
+KERNELSRC_WORK_TREE = ''kernelsrc/''
+
+# read filelist stripping comments and blank lines
+def read_filelist():
+ with open(''kernelsrc/update_kernelsrc.filelist'') as f:
+ filelist = [l for l in (l.split(''#'')[0].strip() for l
in f.readlines()) if l]
+ if filelist:
+ return filelist
+ else:
+ raise Exception(''update_kernelsrc.filelist does not
contain any files'')
+
+# checkout files from kernel git tree
+def checkout_kernelsrc(rev, filelist):
+ cmdline = [''git'', ''--git-dir='' +
KERNELSRC_GIT_DIR, ''--work-tree='' + KERNELSRC_WORK_TREE,
+ ''checkout'', rev, ''--''] +
filelist
+ rc = subprocess.call(cmdline)
+ if rc != 0:
+ raise Exception(''git checkout from kernel source git repo
returned %d'' % rc)
+
+# run a git add on the updated kernelsrc tree to add the changes to the index
+def add_kernelsrc_to_index():
+ # add them to git index of this git tree
+ rc = subprocess.call([''git'', ''add'',
KERNELSRC_WORK_TREE])
+ if rc != 0:
+ raise Exception(''git add %s returned %d'' %
(KERNELSRC_WORK_TREE, rc))
+
+# remove current files from kernelsrc to prepare to receive new ones
+def cleanup_kernelsrc():
+ for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE):
+ for f in filenames:
+ # ignore files named update_kernelsrc.* and README* on root of the
kernelsrc/ tree:
+ if (dirpath == KERNELSRC_WORK_TREE) and
(f.startswith(''update_kernelsrc.'') or
f.startswith(''README'')):
+ continue
+ # otherwise, git rm it
+ filepath = os.path.join(dirpath, f)
+ rc = subprocess.call([''git'',
''rm'', ''-q'', filepath])
+ if rc != 0:
+ raise Exception(''git rm %s returned %d'' %
(filepath, rc))
+ # second pass: remove directories, use a stack to remove them in the right
order
+ stack = []
+ for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE):
+ if dirpath != KERNELSRC_WORK_TREE:
+ stack.append(dirpath)
+ for d in reversed(stack):
+ os.rmdir(d)
+
+# check whether current tree is clean and ready to receive update
+# only allowed change in index is to kernelsrc/update_kernelsrc.filelist
+# changes not in index must be outside of kernelsrc tree
+def check_for_clean_tree():
+ status = subprocess.check_output([''git'',
''status'', ''--porcelain'']).splitlines()
+ for l in status:
+ worktree = l[0]
+ index = l[1]
+ path = l[3:]
+ if (path == os.path.join(KERNELSRC_WORK_TREE,
''update_kernelsrc.filelist'')) and (worktree in ''
M'') and (index in '' M''):
+ # accept changes to the filelist in index and/or worktree
+ continue
+ if (not path.startswith(KERNELSRC_WORK_TREE)) and (worktree ==
''?'') and (index == ''?''):
+ # accept files with unknown status outside of kernelsrc
+ continue
+ # otherwise, abort
+ raise Exception(''current tree is not clean! (e.g.
%s)'' % path)
+
+# resolve tag into revision
+def resolve_tag(tag):
+ cmdline = [''git'', ''--git-dir='' +
KERNELSRC_GIT_DIR, ''rev-parse'', tag + ''^0'']
+ rev = subprocess.check_output(cmdline).strip()
+ return rev
+
+# check whether the new tag is an upgrade from the previous one
+# in other words, if the previous tag is an ancestor of the new one
+def check_upgrade(rev):
+ # previous revision stored in kernelsrc/update_kernelsrc.rev
+ oldrevpath = os.path.join(KERNELSRC_WORK_TREE,
''update_kernelsrc.rev'')
+ if not os.path.exists(oldrevpath):
+ # if it doesn''t exist, then just assume it''s OK for
an upgrade
+ # but return None in order to signal the this is the first checkout
+ return None
+ with open(oldrevpath) as f:
+ oldrev = f.read().strip()
+ # use git merge-base to check what''s the merge base between them
+ # if it''s an upgrade, it will match oldrev
+ cmdline = [''git'', ''--git-dir='' +
KERNELSRC_GIT_DIR, ''merge-base'', oldrev, rev]
+ mergebase = subprocess.check_output(cmdline).strip()
+ if mergebase != oldrev:
+ raise Exception(''new revision %s is not a descendent of
previous revision %s'' % (rev, oldrev))
+ oldtagpath = os.path.join(KERNELSRC_WORK_TREE,
''update_kernelsrc.tag'')
+ # return the old tag (for git commit sample message)
+ try:
+ with open(oldtagpath) as f:
+ oldtag = f.read().strip()
+ return oldtag
+ except OSError:
+ return oldrev
+
+# store tag (and revision) for future query/comparison
+def store_tag(tag, rev):
+ with open(os.path.join(KERNELSRC_WORK_TREE,
''update_kernelsrc.tag''), ''w'') as f:
+ print >>f, tag
+ with open(os.path.join(KERNELSRC_WORK_TREE,
''update_kernelsrc.rev''), ''w'') as f:
+ print >>f, rev
+
+# do it all!
+def main():
+ global KERNELSRC_GIT_DIR
+ if len(sys.argv) != 3:
+ raise Exception(''usage: %s <kernelsrc>
<tag>'' % SCRIPT_NAME)
+ if sys.argv[0] != SCRIPT_NAME:
+ raise Exception(''script must be called as %s from root
directory in btrfs-progs tree'' % SCRIPT_NAME)
+ if not os.path.isdir(''.git''):
+ raise Exception(''script must be called from root directory in
btrfs-progs git tree'')
+ if not os.path.isdir(KERNELSRC_WORK_TREE):
+ raise Exception(''weird, this looks like a git tree but could
not find directory %s'' % KERNELSRC_WORK_TREE)
+ if not os.path.isdir(sys.argv[1]):
+ raise Exception(''first parameter must be the path to the
kernel source directory'')
+ KERNELSRC_GIT_DIR = os.path.join(sys.argv[1], ''.git'')
+ if not os.path.isdir(KERNELSRC_GIT_DIR):
+ raise Exception(''first parameter must be the path to the root
of the kernel git tree'')
+ # all checks so far, let''s get the new tag and see if
it''s good:
+ tag = sys.argv[2]
+ rev = resolve_tag(tag)
+ oldtag = check_upgrade(tag)
+ # all good, let''s check if the tree is clean
+ check_for_clean_tree()
+ # let''s see if we can get the file list ok
+ filelist = read_filelist()
+ # all good! now let''s do the update (remove followed by checkout)
+ cleanup_kernelsrc()
+ checkout_kernelsrc(rev, filelist)
+ # done upgrade, still not in index though...
+ # before adding it to the index, let''s record tag and rev
+ store_tag(tag, rev)
+ # and finally, add to git index
+ add_kernelsrc_to_index()
+ # done!
+ print ''Done upgrading kernelsrc to %s.'' % tag
+ print ''Please test the new tree and commit when ready.''
+ print ''''
+ print ''Use the following command for commit:''
+ if oldtag is None:
+ print '' git commit -em "btrfs-progs: initial checkout
of kernelsrc %s"'' % tag
+ else:
+ print '' git commit -em "btrfs-progs: upgrade kernelsrc
from %s to %s"'' % (oldtag, tag)
+ print ''''
+
+# if called as a script, just run main
+if __name__ == ''__main__'':
+ main()
+
--
1.8.1.2
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html