Hello On Wed, 2008-01-23 at 00:10 +0300, Nikita Danilov wrote:> Hello, > > below is a tentative list of tasks into which WBC effort can be > sub-divided. I also provided a less exact list for the EPOCH component, > and an incomplete list for the STL component. > > WBC tasks are estimated in lines-of-code with the total of (9100 + 3000) > LOC, where LOC is a non-comment, non-whitespace line of the source > file. I believe it is too early to estimate all EPOCH tasks, hopefully > there will be more clear understanding of the situation during next > week. I tried to estimate the EPOCH tasks that are absolutely necessary > during the early stages of WBC development. > > Vladimir is the best person to complete the list of STL tasks, > together with the estimations. > > C-* tasks are for the client, S-* tasks are for the server. > > ESTIMATION > > scope effort > loc > WBC > > C-VFS-MM integration with vfs: inodes, dentries, memory 2500 > pressure. Executing operation effects locally. > > C-ops-caching tracking operations: list vs. fragments. Tracking 2500 > dependencies. > > C-write-out policy deciding when to write-out cached state 1000 > updates, and with what granularity: age, amount, > max-in-flight > > C-dir-pages caching of directory pages and using them for local 1000 > lookups > > C-new-files creation of new files locally 200 > > C-new-objects creation of new objects locally 200 > > C-DLM invoking reintegration on a lock cancel, lock 200 > weighting > > C-data dependencies between cached data and meta-data 200 > > C-IO switching between whole-file mds-based locking and 250 > extent locking > > C-grants unified resource range leasing mechanism 250 > > S-grants unified resource range leasing mechanism 250 > > C-misc sync, fsync, compatibility flag, mount option 200 > > S-misc compatibility flags 100 > > STLComponents of STL (tentatively)>C-policy track usage statistics 250 and use them to decide when to ask for a STL S-policy track usage statistics 250 and use them to decide when to grant a STL S-LDLM new lock type, etc 100 C-S-STL split split STL into STLs of subdirs on callback 1000 Persistent STL store subtree lock on persistent storage 2000 S-path revalidation with STL access to ".." requires special handling 1000 C-filesystem operations this is where requests for subtree lock come from 1000 S-conflicts special cases in lock conflicts resolution, 1500 switching to ordinary locks, lock acquiring under STL, CMD C-WBC interaction with WBC 100 C-S-recovery resending locks after reconnection 100> EPOCHS > > formalization formal reintegration model with "proofs" of recovery > correctness and concurrency control description > > C-reintegration reintegration, including concurrency control, 1000 > integration with ptlrpc > > S-compound implementation of the compound operations on 1000 > the server > > S-reintegration reintegration of batches on the server, thread 1000 > scheduling > > S-undo keeping undo logs > > S-cuts implementation of the CUTs algorithm > > C-gc garbage collection: when to discard cached batches > > S-gc garbage collection: when to discard undo logs > > C-recovery replay, including optional optimistic "pre-replay" > > S-recovery-0 roll-back of the uncommitted epochs > > S-recovery-1 roll-forward from the clients > > EXTERNAL DEPENDENCIES > > ost-fid ost understanding fids, and granting fid sequences > to the clients > Nikita.Best regards, Vladimir
I assume we will use LDLM infrastructure as the basis for STL. Should we be considering any LDLM cleanups first?> -----Original Message----- > From: Vladimir.Saveliev at Sun.COM [mailto:Vladimir.Saveliev at Sun.COM] > Sent: 31 January 2008 7:56 PM > To: Nikita Danilov > Cc: lustre-rabbit-team at sun.com; lustre-discuss at clusterfs.com > Subject: Re: WBC subcomponents. > > Hello > > On Wed, 2008-01-23 at 00:10 +0300, Nikita Danilov wrote: > > Hello, > > > > below is a tentative list of tasks into which WBC effort can be > > sub-divided. I also provided a less exact list for the > EPOCH component, > > and an incomplete list for the STL component. > > > > WBC tasks are estimated in lines-of-code with the total of > (9100 + 3000) > > LOC, where LOC is a non-comment, non-whitespace line of the source > > file. I believe it is too early to estimate all EPOCH > tasks, hopefully > > there will be more clear understanding of the situation during next > > week. I tried to estimate the EPOCH tasks that are > absolutely necessary > > during the early stages of WBC development. > > > > Vladimir is the best person to complete the list of STL tasks, > > together with the estimations. > > > > C-* tasks are for the client, S-* tasks are for the server. > > > > ESTIMATION > > > > scope > effort > > > loc > > WBC > > > > C-VFS-MM integration with vfs: inodes, > dentries, memory 2500 > > pressure. Executing operation effects locally. > > > > C-ops-caching tracking operations: list vs. > fragments. Tracking 2500 > > dependencies. > > > > C-write-out policy deciding when to write-out > cached state 1000 > > updates, and with what granularity: > age, amount, > > max-in-flight > > > > C-dir-pages caching of directory pages and using > them for local 1000 > > lookups > > > > C-new-files creation of new files locally > 200 > > > > C-new-objects creation of new objects locally > 200 > > > > C-DLM invoking reintegration on a lock > cancel, lock 200 > > weighting > > > > C-data dependencies between cached data and > meta-data 200 > > > > C-IO switching between whole-file mds-based > locking and 250 > > extent locking > > > > C-grants unified resource range leasing > mechanism 250 > > > > S-grants unified resource range leasing > mechanism 250 > > > > C-misc sync, fsync, compatibility flag, mount > option 200 > > > > S-misc compatibility flags > 100 > > > > STL > > Components of STL (tentatively) > > > > C-policy track usage statistics 250 > and use them to decide when to ask for a STL > > S-policy track usage statistics 250 > and use them to decide when to grant a STL > > S-LDLM new lock type, etc 100 > > C-S-STL split split STL into STLs of subdirs on callback > 1000 > > Persistent STL store subtree lock on persistent storage > 2000 > > S-path revalidation > with STL access to ".." requires special > handling 1000 > > C-filesystem operations > this is where requests for subtree lock come > from 1000 > > S-conflicts special cases in lock conflicts resolution, > 1500 > switching to ordinary locks, > lock acquiring under STL, CMD > > C-WBC interaction with WBC > 100 > > C-S-recovery resending locks after reconnection > 100 > > > EPOCHS > > > > formalization formal reintegration model with > "proofs" of recovery > > correctness and concurrency control description > > > > C-reintegration reintegration, including concurrency > control, 1000 > > integration with ptlrpc > > > > S-compound implementation of the compound > operations on 1000 > > the server > > > > S-reintegration reintegration of batches on the > server, thread 1000 > > scheduling > > > > S-undo keeping undo logs > > > > S-cuts implementation of the CUTs algorithm > > > > C-gc garbage collection: when to discard > cached batches > > > > S-gc garbage collection: when to discard undo logs > > > > C-recovery replay, including optional optimistic > "pre-replay" > > > > S-recovery-0 roll-back of the uncommitted epochs > > > > S-recovery-1 roll-forward from the clients > > > > EXTERNAL DEPENDENCIES > > > > ost-fid ost understanding fids, and granting > fid sequences > > to the clients > > Nikita. > > Best regards, > Vladimir > >