ReactOS website issues

[Pierre Schweitzer]

Hi all,

This is an acknowledgment and informative email.
Since we migrated the ReactOS website to Drupal, you all have spotted
critical performances issues. We did our best to address most of them in
the following days to ensure smooth browsing on the website.
Unfortunately we could not fix them all.
Even worse, nowadays, ReactOS website faces a kind of "shutdown"
everyday (approximately between 2:45 AM CET and 4:45 AM CET). This is a
really long and critical downtime that affects mainly US users due to
timezones.
Be sure we are aware about it.

Unfortunately, this is not something we can fix from a day to the
following. This is due to our backup policies that completely lock the
MySQL server to ensure consistent backups.
We are currently working on getting this fixed as fast as possible. But
this will require (besides the rest) a two hardware upgrades on our
infrastructure.
Indeed, Drupal stresses our infrastructure much more than old RosCMS.
And we have to fully rethink our web infrastructure and redistribute it,
which we cannot do at the moment due to hardware constraints. Upgrading
our hardware will be in our first steps (obviously, expect downtimes at
that moment). Then, we will finally be able to replicate our MySQL
server to perform load-balancing and prevent shutdowns on backup.

We, the sysadmins team and the website team, agreed on a plan about how
to enhance and strengthen the infrastructure, and thus fix all the
issues we are currently facing. This will take some time, but we are
working together to get ReactOS website back into a decent shape.

Be assured that we are doing our best to make your place nice.
Meanwhile, please forgive us the caused inconvenience.

With my best regards, on behalf of the guys working behind the scene,

-- 
Pierre Schweitzer<pierre at reactos.org>
System Administrator
ReactOS Foundation

[Alex Ionescu]

Hi,

I have to ask, with all the money being spent on hardware & etc, doesn't it
make more sense to just host on AWS or Rackspace?

--
Best regards,
Alex Ionescu

-----Original Message-----
From: ros-general-bounces(a)reactos.org
[mailto:[e-mail]] On Behalf Of Pierre Schweitzer
Sent: Friday, November 15, 2013 12:24 PM
To: ReactOS General List; ReactOS Development List
Subject: [ros-general] ReactOS website issues

Hi all,

This is an acknowledgment and informative email.
Since we migrated the ReactOS website to Drupal, you all have spotted
critical performances issues. We did our best to address most of them in the
following days to ensure smooth browsing on the website.
Unfortunately we could not fix them all.
Even worse, nowadays, ReactOS website faces a kind of "shutdown"
everyday (approximately between 2:45 AM CET and 4:45 AM CET). This is a
really long and critical downtime that affects mainly US users due to
timezones.
Be sure we are aware about it.

Unfortunately, this is not something we can fix from a day to the following.
This is due to our backup policies that completely lock the MySQL server to
ensure consistent backups.
We are currently working on getting this fixed as fast as possible. But this
will require (besides the rest) a two hardware upgrades on our
infrastructure.
Indeed, Drupal stresses our infrastructure much more than old RosCMS.
And we have to fully rethink our web infrastructure and redistribute it,
which we cannot do at the moment due to hardware constraints. Upgrading our
hardware will be in our first steps (obviously, expect downtimes at that
moment). Then, we will finally be able to replicate our MySQL server to
perform load-balancing and prevent shutdowns on backup.

We, the sysadmins team and the website team, agreed on a plan about how to
enhance and strengthen the infrastructure, and thus fix all the issues we
are currently facing. This will take some time, but we are working together
to get ReactOS website back into a decent shape.

Be assured that we are doing our best to make your place nice.
Meanwhile, please forgive us the caused inconvenience.

With my best regards, on behalf of the guys working behind the scene,

--
Pierre Schweitzer<pierre at reactos.org> System Administrator ReactOS
Foundation

[Michael Fritscher]

Hi,

so the backup does need 2 hours? How big is our database?! Thats very,
very long...
A first idea would be mysqlhotcopy, which does the locking etc. for you -
and copy the raw database files (instead of making sql-files which can
take ages). Informations are under
http://dev.mysql.com/doc/refman/5.1/en/mysqlhotcopy.html . It can do
things e.g. skipping the index files which can be rebuilt, flush the logs
etc.

Another approach is using a lvm under the database-files. This way you get
at least a consistent version which are on the HDDs. Yes, you loose all
information which are only in the RAM, but that shouldn't be too much -
und if you need to replay backups you loose always by average 12 hours -
so 5-10 minutes more ore less aren't that important ;)

Best regards,
Michael Fritscher

[Mathis Klooß]

Hi,

mysqlhotcopy is a good idea but will not work with InnoDB,
when your using percona or mariadb you can use "Percona XtraBackup", 
will not work with mysql (from oracle)

http://www.percona.com/software/percona-xtrabackup

best regards
Mathis Klooß

Am 17.11.2013 08:36, schrieb Michael Fritscher:
>   Hi,
>
>  so the backup does need 2 hours? How big is our database?! Thats very,
>  very long...
>  A first idea would be mysqlhotcopy, which does the locking etc. for you -
>  and copy the raw database files (instead of making sql-files which can
>  take ages). Informations are under
>  http://dev.mysql.com/doc/refman/5.1/en/mysqlhotcopy.html . It can do
>  things e.g. skipping the index files which can be rebuilt, flush the logs
>  etc.
>
>  Another approach is using a lvm under the database-files. This way you get
>  at least a consistent version which are on the HDDs. Yes, you loose all
>  information which are only in the RAM, but that shouldn't be too much -
>  und if you need to replay backups you loose always by average 12 hours -
>  so 5-10 minutes more ore less aren't that important ;)
>
>  Best regards,
>  Michael Fritscher
>
>
>
>  _______________________________________________
>  Ros-general mailing list
>  Ros-general(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-general

[Colin Finck]

Thank you Michael and Mathis!

While we already knew about mysqlhotcopy, our tables are completely
InnoDB-based, so this was not an option for us.
Percona XtraBackup looks very good though. We will definitely give it a
try. I just don't find any information on the website that it doesn't
work with vanilla MySQL. Do you have any reference, Mathis?

Cheers,

Colin


Mathis Klooß <admin(a)gunah.eu> wrote:
>   Hi,
>  
>  mysqlhotcopy is a good idea but will not work with InnoDB,
>  when your using percona or mariadb you can use "Percona XtraBackup",
>  will not work with mysql (from oracle)
>  
>  http://www.percona.com/software/percona-xtrabackup
>  
>  best regards
>  Mathis Klooß
>  
>  Am 17.11.2013 08:36, schrieb Michael Fritscher:
> >   Hi,
> >
> >  so the backup does need 2 hours? How big is our database?! Thats very,
> >  very long...
> >  A first idea would be mysqlhotcopy, which does the locking etc. for you -
> >  and copy the raw database files (instead of making sql-files which can
> >  take ages). Informations are under
> >  http://dev.mysql.com/doc/refman/5.1/en/mysqlhotcopy.html . It can do
> >  things e.g. skipping the index files which can be rebuilt, flush the logs
> >  etc.
> >
> >  Another approach is using a lvm under the database-files. This way you
> >  get
> >  at least a consistent version which are on the HDDs. Yes, you loose all
> >  information which are only in the RAM, but that shouldn't be too much -
> >  und if you need to replay backups you loose always by average 12 hours -
> >  so 5-10 minutes more ore less aren't that important ;)
> >
> >  Best regards,
> >  Michael Fritscher
> >
> >
> >
> >  _______________________________________________
> >  Ros-general mailing list
> >  Ros-general(a)reactos.org
> >  http://www.reactos.org/mailman/listinfo/ros-general 
>  
>  
>  _______________________________________________
>  Ros-general mailing list
>  Ros-general(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-general
>