[GSoC] IFS wrapper project proposal draft

[Bryan Donlan]

Hello,

After reading the ReactOS GSoC ideas page, I became interested in the
IFS wrapper driver project idea[1], and would appreciate any comments
on my project proposal before I formally submit it.

First, a bit about me. I am a student in a dual-major program,
studying Computer Science and Japanese at the University of
Massachusetts at Amherst. I have had some experience in kernel
development previously; as part of an internship at a research project
at my University, I wrote some network flow analysis code running in
the Linux kernel. While filesystem work will be a first for me, I
think it will be an interesting leaning experience. I am fluent in
English and Japanese, and am in the UTC-0400 timezone (TZ=US/Eastern).
My freenode username is bd_, and my myReactOS username is bdonlan.

In preparation for writing this proposal, I obtained a copy of the
"Windows NT File System Internals: A Developer's Guide" book from
O'Reilly. Although a bit dated, it has been enourmously helpful in
helping me get an idea of what I'm getting into.

My goal in this project will be to write a library to significantly
simplify writing a NT File System Driver. Although writing a
kernel-mode filesystem will never be as easy as with, eg, FUSE, it
should be possible to abstract away many of the idiosyncracies of the
NT filesystem API, at least. For example, the wrapper library may
handle:
* Automatically queueing asynchronous filesystem operations on a worker thread
* Parsing paths and invoking a FS-supplied traversal function for each
directory element
* Interfacing with the NT cache manager, including flushing data from
the cache when a non-cached operation is performed on a cached file,
and establishing cache maps for the underlying device for a disk-based
file system
* Providing a common implementation of byte range locks and oplocks,
and disabling fast I/O when they are in use
* Caching directory entry lookups
* Performing typical input validation/security checks needed by all NT
filesystems
* Helper tools may also be provided to load pseudo-filesystems
(filesystems with neither network nor disk backing) on an ad-hoc basis

Essentially, the library should make it easy for the filesystem
designer to focus on the actual filesystem, rather than the NT IFS
interfaces - a read call would be able to skip all the preparation and
checks, and directly go to perform the actual I/O.

Major milestones for the project may include:
- Prepare a sketch of the API and callbacks for the IFS helper library
(subject to change if necessary)
- Implement enough of the wrapper library to implement a simple
pseudo-filesystem demonstrating non-cached reads and writes
- Add support for cached reads/writes (interaction with the cache
manager, caching for on-disk metadata)
- Demonstrate a disk-based filesystem without metadata update support
(MINIX FS or FAT?)
- Demonstrate a disk-based filesystem with full read-write support
- If time allows, add support for ancillary features, such as byte
locks, oplocks, and notify watchers.

The interfaces exposed by the filesystem library will of course be
clearly documented. Filesystems needing more advanced support will
also be able to override any IFS callbacks they choose.

I have an existing consulting commitment that will take some of my
time, but I expect to be able to put in 20-30 hours of work per week
into this project. Further, I hereby swear that I have not used nor
seen the source code to any version of the Windows operating system
nor any Microsoft product that may be related to the proposed project
that is under a license incompatible with contribution to ReactOS,
including but not limited to the leaked Windows 2000 source code and
the Windows Research Kernel.

I would appreciate any comments on my proposal before formally
submitting it to the GSoC site.

Thanks,

Bryan Donlan

[1] - http://www.reactos.org/wiki/Google_Summer_of_Code_2011_Ideas#IFS_Wrapper_Dr…

[WaxDragon]

How about iSCSI?

On Sat, Apr 2, 2011 at 6:26 PM, Bryan Donlan <bdonlan(a)gmail.com> wrote:

>   Hello,
>
>  After reading the ReactOS GSoC ideas page, I became interested in the
>  IFS wrapper driver project idea[1], and would appreciate any comments
>  on my project proposal before I formally submit it.
>
>  First, a bit about me. I am a student in a dual-major program,
>  studying Computer Science and Japanese at the University of
>  Massachusetts at Amherst. I have had some experience in kernel
>  development previously; as part of an internship at a research project
>  at my University, I wrote some network flow analysis code running in
>  the Linux kernel. While filesystem work will be a first for me, I
>  think it will be an interesting leaning experience. I am fluent in
>  English and Japanese, and am in the UTC-0400 timezone (TZ=US/Eastern).
>  My freenode username is bd_, and my myReactOS username is bdonlan.
>
>  In preparation for writing this proposal, I obtained a copy of the
>  "Windows NT File System Internals: A Developer's Guide" book from
>  O'Reilly. Although a bit dated, it has been enourmously helpful in
>  helping me get an idea of what I'm getting into.
>
>  My goal in this project will be to write a library to significantly
>  simplify writing a NT File System Driver. Although writing a
>  kernel-mode filesystem will never be as easy as with, eg, FUSE, it
>  should be possible to abstract away many of the idiosyncracies of the
>  NT filesystem API, at least. For example, the wrapper library may
>  handle:
>  * Automatically queueing asynchronous filesystem operations on a worker
>  thread
>  * Parsing paths and invoking a FS-supplied traversal function for each
>  directory element
>  * Interfacing with the NT cache manager, including flushing data from
>  the cache when a non-cached operation is performed on a cached file,
>  and establishing cache maps for the underlying device for a disk-based
>  file system
>  * Providing a common implementation of byte range locks and oplocks,
>  and disabling fast I/O when they are in use
>  * Caching directory entry lookups
>  * Performing typical input validation/security checks needed by all NT
>  filesystems
>  * Helper tools may also be provided to load pseudo-filesystems
>  (filesystems with neither network nor disk backing) on an ad-hoc basis
>
>  Essentially, the library should make it easy for the filesystem
>  designer to focus on the actual filesystem, rather than the NT IFS
>  interfaces - a read call would be able to skip all the preparation and
>  checks, and directly go to perform the actual I/O.
>
>  Major milestones for the project may include:
>  - Prepare a sketch of the API and callbacks for the IFS helper library
>  (subject to change if necessary)
>  - Implement enough of the wrapper library to implement a simple
>  pseudo-filesystem demonstrating non-cached reads and writes
>  - Add support for cached reads/writes (interaction with the cache
>  manager, caching for on-disk metadata)
>  - Demonstrate a disk-based filesystem without metadata update support
>  (MINIX FS or FAT?)
>  - Demonstrate a disk-based filesystem with full read-write support
>  - If time allows, add support for ancillary features, such as byte
>  locks, oplocks, and notify watchers.
>
>  The interfaces exposed by the filesystem library will of course be
>  clearly documented. Filesystems needing more advanced support will
>  also be able to override any IFS callbacks they choose.
>
>  I have an existing consulting commitment that will take some of my
>  time, but I expect to be able to put in 20-30 hours of work per week
>  into this project. Further, I hereby swear that I have not used nor
>  seen the source code to any version of the Windows operating system
>  nor any Microsoft product that may be related to the proposed project
>  that is under a license incompatible with contribution to ReactOS,
>  including but not limited to the leaked Windows 2000 source code and
>  the Windows Research Kernel.
>
>  I would appreciate any comments on my proposal before formally
>  submitting it to the GSoC site.
>
>  Thanks,
>
>  Bryan Donlan
>
>  [1] -
>  http://www.reactos.org/wiki/Google_Summer_of_Code_2011_Ideas#IFS_Wrapper_Dr…
>
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>  


-- 
<+encoded> if you square a unicorn do you get a real animal?

[Bryan Donlan]

On Sat, Apr 2, 2011 at 21:12, WaxDragon <waxdragon(a)gmail.com> wrote:
>   How about iSCSI? 
Although I've never used iSCSI myself, as I understand it, it
essentially just exposes a SCSI device over the network, right? If so,
then handling of iSCSI belongs at the disk device level, not in
individual filesystem drivers; the filesystem doesn't need to know
that its read/write requests will end up being carried over an
Ethernet rather than a SATA cable.

[Aleksey Bragin]

Hi,
thanks for a good proposal! Creating such "library" was my dream for  
quite some time which I never came close to implement due to being  
busy. It would be great if you could achieve that as part of GSoC.


WBR,
Aleksey Bragin.

On Apr 3, 2011, at 2:26 AM, Bryan Donlan wrote:

>   Hello,
>
>  After reading the ReactOS GSoC ideas page, I became interested in the
>  IFS wrapper driver project idea[1], and would appreciate any comments
>  on my project proposal before I formally submit it.
>
>  First, a bit about me. I am a student in a dual-major program,
>  studying Computer Science and Japanese at the University of
>  Massachusetts at Amherst. I have had some experience in kernel
>  development previously; as part of an internship at a research project
>  at my University, I wrote some network flow analysis code running in
>  the Linux kernel. While filesystem work will be a first for me, I
>  think it will be an interesting leaning experience. I am fluent in
>  English and Japanese, and am in the UTC-0400 timezone (TZ=US/Eastern).
>  My freenode username is bd_, and my myReactOS username is bdonlan.
>
>  In preparation for writing this proposal, I obtained a copy of the
>  "Windows NT File System Internals: A Developer's Guide" book from
>  O'Reilly. Although a bit dated, it has been enourmously helpful in
>  helping me get an idea of what I'm getting into.
>
>  My goal in this project will be to write a library to significantly
>  simplify writing a NT File System Driver. Although writing a
>  kernel-mode filesystem will never be as easy as with, eg, FUSE, it
>  should be possible to abstract away many of the idiosyncracies of the
>  NT filesystem API, at least. For example, the wrapper library may
>  handle:
>  * Automatically queueing asynchronous filesystem operations on a  
>  worker thread
>  * Parsing paths and invoking a FS-supplied traversal function for each
>  directory element
>  * Interfacing with the NT cache manager, including flushing data from
>  the cache when a non-cached operation is performed on a cached file,
>  and establishing cache maps for the underlying device for a disk-based
>  file system
>  * Providing a common implementation of byte range locks and oplocks,
>  and disabling fast I/O when they are in use
>  * Caching directory entry lookups
>  * Performing typical input validation/security checks needed by all NT
>  filesystems
>  * Helper tools may also be provided to load pseudo-filesystems
>  (filesystems with neither network nor disk backing) on an ad-hoc basis
>
>  Essentially, the library should make it easy for the filesystem
>  designer to focus on the actual filesystem, rather than the NT IFS
>  interfaces - a read call would be able to skip all the preparation and
>  checks, and directly go to perform the actual I/O.
>
>  Major milestones for the project may include:
>  - Prepare a sketch of the API and callbacks for the IFS helper library
>  (subject to change if necessary)
>  - Implement enough of the wrapper library to implement a simple
>  pseudo-filesystem demonstrating non-cached reads and writes
>  - Add support for cached reads/writes (interaction with the cache
>  manager, caching for on-disk metadata)
>  - Demonstrate a disk-based filesystem without metadata update support
>  (MINIX FS or FAT?)
>  - Demonstrate a disk-based filesystem with full read-write support
>  - If time allows, add support for ancillary features, such as byte
>  locks, oplocks, and notify watchers.
>
>  The interfaces exposed by the filesystem library will of course be
>  clearly documented. Filesystems needing more advanced support will
>  also be able to override any IFS callbacks they choose.
>
>  I have an existing consulting commitment that will take some of my
>  time, but I expect to be able to put in 20-30 hours of work per week
>  into this project. Further, I hereby swear that I have not used nor
>  seen the source code to any version of the Windows operating system
>  nor any Microsoft product that may be related to the proposed project
>  that is under a license incompatible with contribution to ReactOS,
>  including but not limited to the leaked Windows 2000 source code and
>  the Windows Research Kernel.
>
>  I would appreciate any comments on my proposal before formally
>  submitting it to the GSoC site.
>
>  Thanks,
>
>  Bryan Donlan
>
>  [1] - http://www.reactos.org/wiki/ 
>  Google_Summer_of_Code_2011_Ideas#IFS_Wrapper_Driver
>
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev