RosFS

[Rick Langschultz]

For those of you who have heard nothing of my plans to write a RosFS
service for ReactOS, here are some details about my plan.

RosFS

RosFS is a small service running on system initialization by default
that indexes files, folders, users, devices, and network resources can
hold metadata via MySQL, sleepycat DB, TCL, Apache, and Perl; which will
all be embedded within the service itself. RosFS will hold encrypted
resources based off of a key that is created with the first startup of
ReactOS, it changes every 30 days to ensure security. The RosFS can be
accessed online via the built in web server with can be an encrypted
connection - SSL, or TLS. User ACLs can be set on a machine level or a
domain level. RosFS will support a SQL-like querying language that can
be written easily - it will have triggers, procedures, functions, and
saved queries - which are all logged. Triggers will be supplied by a
TCL-like language with features of Perl.  An service will provide users
with current system updates, patches, etc stored on a domain server -
this option will only be installed on a powerful server with pretty good
hardware - ie: not for home use!!!

A few problems I know that will arise in developing such a project are:
1) Size
2) Security
3) Where to keep the files on the system
4) The RosFS query language will be trouble to write and maintain
depending on if a user uses such Databases like ORACLE, MySQL, MSSQL,
etc.
5) Explorer Integration - Will Explorer.exe need to be rewritten to
allow the addition of Metadata structures, etc.
6) Web integration - Perl, PHP, TCL, CGI, ISAPI???

Please send me comments and concerns about the RosFS project.

[Martin Fuchs]

Hello,
> RosFS
>
> RosFS is a small service running on system initialization by default
> that indexes files, folders, users, devices, and network resources can
> hold metadata via MySQL, sleepycat DB, TCL, Apache, and Perl; which will
> all be embedded within the service itself. RosFS will hold encrypted

Sounds like quite much work, what you are proposing.
> A few problems I know that will arise in developing such a project are:
> Size
> Security
> Where to keep the files on the system

May be somewhere in %SystemRoot%\IndexData\ if you want
to use a DB like MySQL?
However MS uses another approach. It stores this info in the hidden
"X:\System Volume Information" folders separated by drive.
It seems, they also use the NTFS 5-feature of change logs to update the
indexes. What about trying to do it the same way?
This would have the advantage of avoiding synchronisation problems.
For example: Imagine one removes a drive from the system, and puts
it in another computer. If the metadata is located directly on the drive,
you don't have to update all index information. Or imagine, you have
got two installations of ROS on one system. This way both can share
the same indexes.
> The RosFS query language will be trouble to write and maintain
>
> depending on if a user uses such Databases like ORACLE, MySQL, MSSQL,
> etc.

Well, this language remembers me of the OQL language used in projects like
Castor or Hibernate. Just another layer on top of SQL to become independent
from the actual database used.
Did you know, WMI also uses a SQL dialect to query services, drive inforation,
and whatever else...? May be better to write a WMI implementation for ROS
instead of re-inventing the wheel.
> Explorer Integration - Will Explorer.exe need to be rewritten to
>
> allow the addition of Metadata structures, etc.

Not rewritten - maybe extended to be able to query the additional information.
> Web integration - Perl, PHP, TCL, CGI, ISAPI???
> Please send me comments and concerns about the RosFS project.

Regards,

Martin

[Royce Mitchell III]

Martin Fuchs wrote:
> > Explorer Integration - Will Explorer.exe need to be rewritten to
> >
> > allow the addition of Metadata structures, etc.
>
> Not rewritten - maybe extended to be able to query the additional information.

call me silly, but can't this be done via a shell extension without 
having to touch explorer?
-- 
------------------------
http://wecanstopspam.org

[Thomas Weidenmueller]

Royce Mitchell III wrote:
> Martin Fuchs wrote:
>
> call me silly, but can't this be done via a shell extension without 
> having to touch explorer?

It can be done once wine supports them.

Thomas

[Martin Fuchs]

> > > Explorer Integration - Will Explorer.exe need to be rewritten to
> > >
> > > allow the addition of Metadata structures, etc.
> >
> > Not rewritten - maybe extended to be able to query the additional
>
> information.
> >
>
> call me silly, but can't this be done via a shell extension without 
> having to touch explorer?

Depends on what you are trying to do exactly.
And "extending explorer" includes things like touching shell32.dll
for me. You can't look at them separately for most features,
they are just too much integrated.

[Robert Köpferl]

Sorry Guys, I am a newbe to all this WinFS stuff.

But one thing, I know for sure:
I dislike the imagination of having a Database which holds data about 
objects in a file system.  It is like having two relations (R1: 
Directory, R2: DB-Table) which are formally 1:1 but there's nothing 
which keeps their integrity.
What happenes, if I boot an OS wich just works on the Filesystem?
Do I need a 'chkdsk+db' then?

[Magnus Olsen]

Hi
here is what I think about the hole rosfs.
after I have read about it. do not feel offended. 
but some points I feel it some part are not good
idea.

Should this not need alot of cpu power and memory use ?
if it need alot of memory most of i486 does only have 32MB 
today reactos can run on a i486 with 32MB with out any problem. 
but I feel in this way it need alot more memory that 32MB.

the second problem I feel it need alot of harddisk space
and it will need alot of harddisk space, that goes wast 
I want a os that does not wast harddisk space.

next problem

I do not like the idea have a sql server install to use it. 
it will need alot of cpu power it will be to havey for i486

And last I do not want a someting that can be using for hacking 
into my computer more easy. I feel rosfs will create alot of hole.
I do not feel safe with this idea.  and it maby will make most 
commo people unsure if reactos are safe or not.

BestReagds
Magnus Olsen

----- Original Message ----- 
  From: Rick Langschultz 
  To: [e-mail] 
  Sent: Wednesday, September 29, 2004 2:43 PM
  Subject: [ros-dev] RosFS

For those of you who have heard nothing of my plans to write a RosFS service for ReactOS, here are some details about my plan.

RosFS

RosFS is a small service running on system initialization by default that indexes files, folders, users, devices, and network resources can hold metadata via MySQL, sleepycat DB, TCL, Apache, and Perl; which will all be embedded within the service itself. RosFS will hold encrypted resources based off of a key that is created with the first startup of ReactOS, it changes every 30 days to ensure security. The RosFS can be accessed online via the built in web server with can be an encrypted connection - SSL, or TLS. User ACLs can be set on a machine level or a domain level. RosFS will support a SQL-like querying language that can be written easily - it will have triggers, procedures, functions, and saved queries - which are all logged. Triggers will be supplied by a TCL-like language with features of Perl.  An service will provide users with current system updates, patches, etc stored on a domain server - this option will only be installed on a powerful server with pretty good hardware - ie: not for home use!!!

A few problems I know that will arise in developing such a project are:
  1) Size

2) Security
  3) Where to keep the files on the system
  4) The RosFS query language will be trouble to write and maintain depending on if a user uses such Databases like ORACLE, MySQL, MSSQL, etc.
  5) Explorer Integration - Will Explorer.exe need to be rewritten to allow the addition of Metadata structures, etc.
  6) Web integration - Perl, PHP, TCL, CGI, ISAPI???

Please send me comments and concerns about the RosFS project.

------------------------------------------------------------------------------

_______________________________________________
  Ros-dev mailing list
  [e-mail]
  http://reactos.com:8080/mailman/listinfo/ros-dev

[Jasper van de Gronde]

Magnus Olsen wrote:
> Should this not need alot of cpu power and memory use ?
> if it need alot of memory most of i486 does only have 32MB 
> today reactos can run on a i486 with 32MB with out any problem. 
> but I feel in this way it need alot more memory that 32MB.

Anything will of course use cpu power and memory. As for exact figures, 
that would be guesswork, as it depends heavily on how the system is 
implemented, and I wouldn't know of any benchmarks of similar systems on 
such machines.
> the second problem I feel it need alot of harddisk space
> and it will need alot of harddisk space, that goes wast 
> I want a os that does not wast harddisk space.

That depends entirely on how the data is stored. As an example, RDF (I 
don't know what RosFS will use, but I know RDF well and it's just an 
example of how storage methods can matter) can be stored in numerous 
ways, one way is using RDF/XML, which looks a bit like this (omitting 
namespace declarations and such for clarity):
   rdf:RDF
     <rdf:Description rdf:about="some_url">
       dc:titleSome title</dc:title>
       dc:descriptionA very interesting description.</dc:description>
     </rdf:Description>
   </rdf:RDF>
Another way is using Notation3:
   <some_url> dc:title "Some title" ;
              dc:description "A very interesting description." .
As you can see this last representation is quite a bit less verbose than 
the previous representation.
> next problem
>
> I do not like the idea have a sql server install to use it. 
> it will need alot of cpu power it will be to havey for i486

I agree the system should be somewhat scalable, but as Rick mentioned it 
will probably be an embedded server, which is a bit more light weight. 
And there is also a good chance a full-blown sql server isn't needed (at 
least not for everyone) and could be replaced by a specialized database 
engine.
For example, the server I'm using for some RDF data at the moment is 
only about 900KB and that includes parsers for a number of RDF 
representations and Curl. Depending on what the minimum requirements of 
RosFS will be it's server probably wouldn't have to be much larger than 
that. A small server like this would mostly use memory as cache, and 
that can be controlled depending on the needs. And the CPU requirements 
depend on how much data it needs to handle and usually aren't that bad 
(MySQL is quite fast for example, and even a relatively unoptimized 
server like I use spits out records faster than you can imagine, over a 
TCP/IP connection that is).

[Jasper van de Gronde]

Rick Langschultz wrote:

I'd first like to state I'm a big proponent of any advances in metadata 
handling and think it's a great idea to do something like this for ROS 
(and possibly other systems).

First off, what kind of data model are you thinking of? Something 
graph-like (like RDF), something more tree-like (no multiple-parents), 
something even flatter, or perhaps something entirely different. Also, 
what kind of (meta)data are you thinking of storing? Just "system" data 
(things like modified times), just "user" data (title/description) or 
just anything one would like to store?
> RosFS is a small service running on system initialization by default
> that indexes files, folders, users, devices, and network resources can
> hold metadata via MySQL, sleepycat DB, TCL, Apache, and Perl; which will
> all be embedded within the service itself.

Could you explain in more detail how you envision the structure of this 
service? For example, why MySQL AND Sleepycat DB? Is it meant to have a 
way to attach different database backends relatively easily? (perhaps 
not necessarily at runtime at first, but at least at compiletime)
Why TCL? Or is this just one possible scripting language you plan on 
supporting?
What storage format will be used? And what will applications use to get 
data from the storage?
> RosFS will hold encrypted
> resources based off of a key that is created with the first startup of
> ReactOS, it changes every 30 days to ensure security.

What do you mean by resources in this case? And why do they need to be 
encrypted? Is the encryption meant to be used as some form of access 
control?
> The RosFS can be
> accessed online via the built in web server with can be an encrypted
> connection - SSL, or TLS.

What kind of interface will the web server expose? Something like 
WebDAV? Multiple interfaces, depending on plugins of some sort?
> User ACLs can be set on a machine level or a
> domain level.

I assume you mean a domain as used with Windows networking.
What will the ACLs protect, the entire database or individual records? 
If they protect the entire database, what will be used to determine who 
is allowed to change/read/create/etc. values in the database.
> RosFS will support a SQL-like querying language that can
> be written easily - it will have triggers, procedures, functions, and
> saved queries - which are all logged.

Might I suggest writing a relatively flexible query engine which will 
allow you to use several query languages with it. One possible reason 
for this could be that you'd like to support both very compact query 
languages and very verbose query languages. Another reason could be that 
you'd want to change the query language later on or support a new great 
query language that doesn't exist yet (there are a lot of query 
languages for these kinds of purposes in development).
> Triggers will be supplied by a
> TCL-like language with features of Perl.  An service will provide users
> with current system updates, patches, etc stored on a domain server -
> this option will only be installed on a powerful server with pretty good
> hardware - ie: not for home use!!!

With "this" option I assume you're referring to the patch update stuff?
> A few problems I know that will arise in developing such a project are:
> Size

Of what? The application itself or the data it stores? The application 
itself could indeed grow big, but it could be kept somewhat limited by 
modularizing it in such a way that not everyone would need all parts.
> Security

That depends on what kind of accesses you allow, if you just allow 
access to the DB from the "local" computer and make sure everything is 
properly protected by ACLs or something like that than it should be 
reasonably safe.
> Where to keep the files on the system

Martin Fuchs's suggestion of splitting it across drives seems like a 
reasonable suggestion. And I don't know what change logs are in NTFS, 
but if they're anything like transactions I'd recommend them.
> The RosFS query language will be trouble to write and maintain
>
> depending on if a user uses such Databases like ORACLE, MySQL, MSSQL,
> etc.

That's another reason to make a clear distinction between the query 
parser and query engine, that way you'd be able to write the parse code 
once and then write different backends for different databases. You may 
want to have a look at what the Redland project did (I'm not convinced 
their implementation is ideal, but it shows a possible way to go):
http://librdf.org/
> Explorer Integration - Will Explorer.exe need to be rewritten to
>
> allow the addition of Metadata structures, etc.

That depends on what you want to do (and what the current ROS explorer 
supports, as I have no idea what it does support at the moment). The MS 
Windows explorer supports quite a few extensions (especially in WXP) 
meant for handling metadata (a simple example is a column handler, which 
can add columns to explorer's details view).
> Web integration - Perl, PHP, TCL, CGI, ISAPI???

If you mean language bindings, sure, but I'd first get the basic system 
more or less working before even thinking of something like this.

I'd also like to mention a project I'm working on, MDDB, which at least 
has some similarities to what you're trying to do. It also has some 
important differences (it has a clear distinction between a file and a 
document for example, which is probably not entirely what you had in 
mind). The current site is available at:
http://home.hccnet.nl/th.v.d.gronde/dev/mddb/

In general you may want to look at some RDF based projects out there, 
there are some great things being done with RDF at the moment and if 
nothing else it might at least give you some ideas.

[Martin Fuchs]

> Martin Fuchs's suggestion of splitting it across drives seems like a 
> reasonable suggestion. And I don't know what change logs are in NTFS, 
> but if they're anything like transactions I'd recommend them.

Using NTFS changelogs is a way to minimize the system load of permanently
updating the index information.
To say it in simple words: You ask the system periodicly, which files have
been changed or moved and only update the index information for this changes.
Any other files and directories remain untouched after an initial full index update.

You can read a bit more about this in section "Integration with Windows 2000 Comonent" at:
http://www.microsoft.com/windows2000/techinfo/reskit/en-us/default.asp?url=/...

[K McI]

Rick Langschultz wrote:
> For those of you who have heard nothing of my plans to write a RosFS 
> service for ReactOS, here are some details about my plan.

So, is this just a service, or is it a filesystem itself, like NTFS? Both?
> RosFS
>
> RosFS is a small service running on system initialization by default

So what happens if the service is disabled and the system restarted?
> that indexes files, folders, users, devices, and network resources can 
> hold metadata via MySQL, sleepycat DB, TCL, Apache, and Perl; which will 
> all be embedded within the service itself.

I hope they're slimmed down alot, having a a huge svc just for the file 
system doesn't sound too good.
> RosFS will hold encrypted 
> resources based off of a key that is created with the first startup of 
> ReactOS, it changes every 30 days to ensure security.

What will be encrypted, how and, why? (NEWBIE)
> The RosFS can be 
> accessed online via the built in web server with can be an encrypted 
> connection – SSL, or TLS.

Meaning that all machines will have :80 or something open for remote FS 
access? Or what? (SEMINEWBIE)
> (snipp)
> An service will provide users service with current system updates, patches, etc stored on a domain server – 
> this option will only be installed on a powerful server with pretty good 
> hardware – ie: not for home use!!!

When ROS has good netwking, why not just make RLUC (React Local Update 
Cache (Think MS SUS)) for ALL patches/updates/etc/etc
> A few problems I know that will arise in developing such a project are:
> Size

The roll Apache/MySQL/etc into one will certainly present a problem, I 
have a feeling.
> Security

How do you mean?
> Where to keep the files on the system

MF seems to have solved this
> The RosFS query language will be trouble to write and maintain
>
> depending on if a user uses such Databases like ORACLE, MySQL, MSSQL, etc.
> 5) Explorer Integration – Will Explorer.exe need to be rewritten to 
> allow the addition of Metadata structures, etc.

If it is, what will that do/accomplish?
> Web integration – Perl, PHP, TCL, CGI, ISAPI???

What will that do? Other then open up many holes/avenues of attack? (It 
might be the greatest thing since sliced bread, but after many Windows 
attacks, opening ports to system services seems to be bad idea)
> Please send me comments and concerns about the RosFS project.

OK, did so.

-uniQ

[Lucio Diaz]

--- K McI [e-mail] escribió:
> Rick Langschultz wrote:
> > For those of you who have heard nothing of my
>
> plans to write a RosFS
> > service for ReactOS, here are some details about
>
> my plan.
>
> So, is this just a service, or is it a filesystem
> itself, like NTFS? Both?
> > RosFS
> >
> > RosFS is a small service running on system
>
> initialization by default
>
> So what happens if the service is disabled and the
> system restarted?
> > that indexes files, folders, users, devices, and
>
> network resources can
> > hold metadata via MySQL, sleepycat DB, TCL,
>
> Apache, and Perl; which will
> > all be embedded within the service itself.
>
> I hope they're slimmed down alot, having a a huge
> svc just for the file 
> system doesn't sound too good.
> > RosFS will hold encrypted 
> > resources based off of a key that is created with
>
> the first startup of
> > ReactOS, it changes every 30 days to ensure
>
> security.
>
> What will be encrypted, how and, why? (NEWBIE)
> > The RosFS can be 
> > accessed online via the built in web server with
>
> can be an encrypted
> > connection ? SSL, or TLS.
>
> Meaning that all machines will have :80 or something
> open for remote FS 
> access? Or what? (SEMINEWBIE)
> > (snipp)
> > An service will provide users service with current
>
> system updates, patches, etc stored on a domain
> server ?
> > this option will only be installed on a powerful
>
> server with pretty good
> > hardware ? ie: not for home use!!!
>
> When ROS has good netwking, why not just make RLUC
> (React Local Update 
> Cache (Think MS SUS)) for ALL
> patches/updates/etc/etc
> > A few problems I know that will arise in
>
> developing such a project are:
> > Size
>
> The roll Apache/MySQL/etc into one will certainly
> present a problem, I 
> have a feeling.
> > Security
>
> How do you mean?
> > Where to keep the files on the system
>
> MF seems to have solved this
> > The RosFS query language will be trouble to
>
> write and maintain
> > depending on if a user uses such Databases like
>
> ORACLE, MySQL, MSSQL, etc.
> > Explorer Integration ? Will Explorer.exe need
>
> to be rewritten to
> > allow the addition of Metadata structures, etc.
>
> If it is, what will that do/accomplish?
> > Web integration ? Perl, PHP, TCL, CGI, ISAPI???
>
> What will that do? Other then open up many
> holes/avenues of attack? (It 
> might be the greatest thing since sliced bread, but
> after many Windows 
> attacks, opening ports to system services seems to
> be bad idea)
> > Please send me comments and concerns about the
>
> RosFS project.
>
> OK, did so.
>     -uniQ

I agree completly, if i want a web server i will
install a web server, but i dont want to have to eat a
web server i wont use, a web server,myslql or any
other Database (including metadata).

Lucio Diaz.

______________________________________________
Renovamos el Correo Yahoo!: ¡100 MB GRATIS!
Nuevos servicios, más seguridad
http://correo.yahoo.es

[Rick Langschultz]

Lucio Diaz

I agree completly, if i want a web server i will
    install a web server, but i dont want to have to eat a
    web server i wont use, a web server,myslql or any
    other Database (including metadata).

Well then don't install ROSFS, it will be an optional addition to your
system. You do not have to install it; you don't have to even read this
email if you don't want to. I just thought it would be nice for home
users to use this feature when they want to search their files. Encrypt
files, etc.

I might even consider changing the name because it will work in Windows
2000 and Xp Home and Professional alike.