I need help fixing rbuild and testing builds on *nix machines!

[Eric Kohl]

Hi!

In order to fix bug #2482 I needed to make inflib Unicode-aware. My 
local ReactOS setup is curently able to build Boot-CD and Live-CD using 
Unicode hive*.inf files.

Rbuild is the only tool that still uses the original inflib. Could 
someone who knows more about C++ and STL than I do have a look at rbuild 
and make it work with the new inflib?

I can provide a patch that includes newinflib (Unicode-aware inflib) and 
  my changes to usetup and mkhive.

I also need someone who can test the whole patch on a *nix machine. Any 
volunters?

Regards,
Eric

[Aleksey Bragin]

Hi Eric,
please attach the patch here, if it's under 250kb.

WBR,
Aleksey.

On Apr 23, 2010, at 1:42 PM, Eric Kohl wrote:
> Hi!
>
> In order to fix bug #2482 I needed to make inflib Unicode-aware. My  
> local ReactOS setup is curently able to build Boot-CD and Live-CD  
> using Unicode hive*.inf files.
>
> Rbuild is the only tool that still uses the original inflib. Could  
> someone who knows more about C++ and STL than I do have a look at  
> rbuild and make it work with the new inflib?
>
> I can provide a patch that includes newinflib (Unicode-aware  
> inflib) and  my changes to usetup and mkhive.
>
> I also need someone who can test the whole patch on a *nix machine.  
> Any volunters?
>
> Regards,
> Eric

[Eric Kohl]

Hi Alesey,

the patch isnt't that large so I attached it.

Here is a description about how to build and test the new inflib:

The modification consists of three parts:
1) newinflib.zip: Contains the newinflib library and should be extracted 
  in the lib directory.
2) newinflib.diff: Contains the required changes to the existing files 
and should be applied to the projects root directory.
3) rbuild.diff: Contains the changes to build rbuild using newinflib 
instead of inflib. This file should be applied to the projects root 
directory.

After extracting newinflib.zip and applying newinflib.diff usetup and 
mkhive will use newinflib instead of inflib. You can test this by 
booting a newly built BootCD or LiveCD after converting a 
boot/bootdata/hive*_i386.inf file to Unicode.

After applying rbuild.diff rbuild will fail to build because 
syssetupgenerator.cpp needs to be converted to use the Unicode API.

In order to test if rbuild handles Unicode files correctly you can 
convert media/inf/syssetup.inf.tpl to Unicode and run a full rebuild. 
The generated file output-i386/media/inf/syssetup.inf should be a 
Unicode file.

Please let me know if you have any further questions!

Regards
Eric

Aleksey Bragin wrote:
> Hi Eric,
> please attach the patch here, if it's under 250kb.
>
> WBR,
> Aleksey.
>
> On Apr 23, 2010, at 1:42 PM, Eric Kohl wrote:
> > Hi!
> >
> > In order to fix bug #2482 I needed to make inflib Unicode-aware. My 
> > local ReactOS setup is curently able to build Boot-CD and Live-CD 
> > using Unicode hive*.inf files.
> >
> > Rbuild is the only tool that still uses the original inflib. Could 
> > someone who knows more about C++ and STL than I do have a look at 
> > rbuild and make it work with the new inflib?
> >
> > I can provide a patch that includes newinflib (Unicode-aware inflib) 
> > and  my changes to usetup and mkhive.
> >
> > I also need someone who can test the whole patch on a *nix machine. 
> > Any volunters?
> >
> > Regards,
> > Eric
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev

Index: base/setup/usetup/usetup.rbuild
===================================================================
--- base/setup/usetup/usetup.rbuild	(revision 46987)
+++ base/setup/usetup/usetup.rbuild	(working copy)
@@ -4,10 +4,12 @@
    <bootstrap installbase="$(CDOUTPUT)/system32" nameoncd="smss.exe" />
    <include base="usetup">.</include>
    <include base="zlib">.</include>
-	<include base="inflib">.</include>
+	<!--include base="inflib">.</include-->
+	<include base="newinflib">.</include>
    <include base="ReactOS">include/reactos/drivers</include>
    <library>zlib</library>
-	<library>inflib</library>
+	<!--library>inflib</library-->
+	<library>newinflib</library>
    <library>ext2lib</library>
    <library>vfatlib</library>
    <library>ntdll</library>
Index: lib/lib.rbuild
===================================================================
--- lib/lib.rbuild	(revision 46987)
+++ lib/lib.rbuild	(working copy)
@@ -37,6 +37,9 @@
    <directory name="lsalib">
    	<xi:include href="lsalib/lsalib.rbuild" />
    </directory>
+	<directory name="newinflib">
+		<xi:include href="newinflib/inflib.rbuild" />
+	</directory>
    <directory name="nls">
    	<xi:include href="nls/nls.rbuild" />
    </directory>
Index: tools/mkhive/mkhive.rbuild
===================================================================
--- tools/mkhive/mkhive.rbuild	(revision 46987)
+++ tools/mkhive/mkhive.rbuild	(working copy)
@@ -1,13 +1,15 @@
 <?xml version="1.0"?>
 <!DOCTYPE module SYSTEM "../../tools/rbuild/project.dtd">
 <module name="mkhive" type="buildtool">
-	<include base="inflibhost">.</include>
+	<!--include base="inflibhost">.</include-->
+	<include base="newinflibhost">.</include>
    <include base="cmlibhost">.</include>
    <include base="zlibhost">.</include>
    <include base="rtl">.</include>
    <define name="MKHIVE_HOST" />
    <compilerflag compilerset="gcc">-fshort-wchar</compilerflag>
-	<library>inflibhost</library>
+	<!--library>inflibhost</library-->
+	<library>newinflibhost</library>
    <library>cmlibhost</library>
    <library>host_wcsfuncs</library>
    <file>binhive.c</file>
Index: tools/mkhive/reginf.c
===================================================================
--- tools/mkhive/reginf.c	(revision 46987)
+++ tools/mkhive/reginf.c	(working copy)
@@ -51,34 +51,34 @@
 /* FUNCTIONS ****************************************************************/

static BOOL
-GetRootKey (PCHAR Name)
+GetRootKey (PWCHAR Name)
 {
-	if (!strcasecmp (Name, "HKCR"))
+	if (!wcsicmp (Name, L"HKCR"))
    {
-		strcpy (Name, "\Registry\Machine\SOFTWARE\Classes\");
+		wcscpy (Name, L"\Registry\Machine\SOFTWARE\Classes\");
    	return TRUE;
    }

-	if (!strcasecmp (Name, "HKCU"))
+	if (!wcsicmp (Name, L"HKCU"))
    {
-		strcpy (Name, "\Registry\User\.DEFAULT\");
+		wcscpy (Name, L"\Registry\User\.DEFAULT\");
    	return TRUE;
    }

-	if (!strcasecmp (Name, "HKLM"))
+	if (!wcsicmp (Name, L"HKLM"))
    {
-		strcpy (Name, "\Registry\Machine\");
+		wcscpy (Name, L"\Registry\Machine\");
    	return TRUE;
    }

-	if (!strcasecmp (Name, "HKU"))
+	if (!wcsicmp (Name, L"HKU"))
    {
-		strcpy (Name, "\Registry\User\");
+		wcscpy (Name, L"\Registry\User\");
    	return TRUE;
    }

#if 0
-	if (!strcasecmp (Name, "HKR"))
+	if (!wcsicmp (Name, L"HKR"))
    	return FALSE;
 #endif

@@ -94,19 +94,19 @@
 static VOID
 AppendMultiSzValue (
    IN HKEY KeyHandle,
-	IN PCHAR ValueName,
-	IN PCHAR Strings,
+	IN PWCHAR ValueName,
+	IN PWCHAR Strings,
    IN SIZE_T StringSize)
 {
    SIZE_T Size;
    ULONG Type;
    size_t Total;
-	PCHAR Buffer;
-	PCHAR p;
+	PWCHAR Buffer;
+	PWCHAR p;
    size_t len;
    LONG Error;

-	Error = RegQueryValueExA (
+	Error = RegQueryValueExW (
    	KeyHandle,
    	ValueName,
    	NULL,
@@ -117,11 +117,11 @@
        (Type != REG_MULTI_SZ))
    	return;

-	Buffer = malloc (Size + StringSize);
+	Buffer = malloc (Size + (StringSize * sizeof(WCHAR)));
    if (Buffer == NULL)
    	return;

-	Error = RegQueryValueExA (
+	Error = RegQueryValueExW (
    	KeyHandle,
    	ValueName,
    	NULL,
@@ -135,25 +135,25 @@
    Total = Size;
    while (*Strings != 0)
    {
-		len = strlen (Strings) + 1;
+		len = wcslen (Strings) + 1;

-		for (p = Buffer; *p != 0; p += strlen (p) + 1)
-			if (!strcasecmp (p, Strings))
+		for (p = Buffer; *p != 0; p += wcslen (p) + 1)
+			if (!wcsicmp (p, Strings))
    			break;

if (*p == 0)  /* not found, need to append it */
    	{
-			memcpy (p, Strings, len);
+			memcpy (p, Strings, len * sizeof(WCHAR));
    		p[len] = 0;
-			Total += len;
+			Total += len * sizeof(WCHAR);
    	}
    	Strings += len;
    }

if (Total != Size)
    {
-		DPRINT ("setting value %s to %s\n", ValueName, Buffer);
-		RegSetValueExA (
+		DPRINT ("setting value %S to %S\n", ValueName, Buffer);
+		RegSetValueExW (
    		KeyHandle,
    		ValueName,
    		0,
@@ -175,11 +175,11 @@
 static BOOL
 do_reg_operation(
    IN HKEY KeyHandle,
-	IN PCHAR ValueName,
+	IN PWCHAR ValueName,
    IN PINFCONTEXT Context,
    IN ULONG Flags)
 {
-	CHAR EmptyStr = (CHAR)0;
+	WCHAR EmptyStr = (WCHAR)0;
    ULONG Type;
    ULONG Size;
    LONG Error;
@@ -188,11 +188,11 @@
    {
    	if (ValueName)
    	{
-				RegDeleteValueA (KeyHandle, ValueName);
+			RegDeleteValueW (KeyHandle, ValueName);
    	}
    	else
    	{
-			RegDeleteKeyA (KeyHandle, NULL);
+			RegDeleteKeyW (KeyHandle, NULL);
    	}

return TRUE;
@@ -203,7 +203,7 @@

if (Flags & (FLG_ADDREG_NOCLOBBER | FLG_ADDREG_OVERWRITEONLY))
    {
-		Error = RegQueryValueExA (
+		Error = RegQueryValueExW (
    		KeyHandle,
    		ValueName,
    		NULL,
@@ -253,7 +253,7 @@
    if (!(Flags & FLG_ADDREG_BINVALUETYPE) ||
        (Type == REG_DWORD && InfHostGetFieldCount (Context) == 5))
    {
-		PCHAR Str = NULL;
+		WCHAR *Str = NULL;

if (Type == REG_MULTI_SZ)
    	{
@@ -262,7 +262,7 @@

if (Size)
    		{
-				Str = malloc (Size);
+				Str = malloc (Size * sizeof(WCHAR));
    			if (Str == NULL)
    				return FALSE;

@@ -292,7 +292,7 @@

if (Size)
    		{
-				Str = malloc (Size);
+				Str = malloc (Size * sizeof(WCHAR));
    			if (Str == NULL)
    				return FALSE;

@@ -302,11 +302,11 @@

if (Type == REG_DWORD)
    	{
-			ULONG dw = Str ? strtoul (Str, NULL, 0) : 0;
+			ULONG dw = Str ? wcstoul (Str, NULL, 0) : 0;

-			DPRINT("setting dword %s to %x\n", ValueName, dw);
+			DPRINT("setting dword %S to %x\n", ValueName, dw);

-			RegSetValueExA (
+			RegSetValueExW (
    			KeyHandle,
    			ValueName,
    			0,
@@ -316,27 +316,27 @@
    	}
    	else
    	{
-			DPRINT("setting value %s to %s\n", ValueName, Str);
+			DPRINT("setting value %S to %S\n", ValueName, Str);

if (Str)
    		{
-				RegSetValueExA (
+				RegSetValueExW (
    				KeyHandle,
    				ValueName,
    				0,
    				Type,
    				(PVOID)Str,
-					(ULONG)Size);
+					(ULONG)Size * sizeof(WCHAR));
    		}
    		else
    		{
-				RegSetValueExA (
+				RegSetValueExW (
    				KeyHandle,
    				ValueName,
    				0,
    				Type,
    				(PVOID)&EmptyStr,
-					(ULONG)sizeof(CHAR));
+					(ULONG)sizeof(WCHAR));
    		}
    	}
    	free (Str);
@@ -358,7 +358,7 @@
    		InfHostGetBinaryField (Context, 5, Data, Size, NULL);
    	}

-		RegSetValueExA (
+		RegSetValueExW (
    		KeyHandle,
    		ValueName,
    		0,
@@ -378,10 +378,10 @@
  * Called once for each AddReg and DelReg entry in a given section.
  */
 static BOOL
-registry_callback (HINF hInf, PCHAR Section, BOOL Delete)
+registry_callback (HINF hInf, PWCHAR Section, BOOL Delete)
 {
-	CHAR Buffer[MAX_INF_STRING_LENGTH];
-	PCHAR ValuePtr;
+	WCHAR Buffer[MAX_INF_STRING_LENGTH];
+	PWCHAR ValuePtr;
    ULONG Flags;
    size_t Length;

@@ -403,11 +403,11 @@
    		continue;

/* get key */
-		Length = strlen (Buffer);
+		Length = wcslen (Buffer);
    	if (InfHostGetStringField (Context, 2, Buffer + Length, MAX_INF_STRING_LENGTH - (ULONG)Length, NULL) != 0)
    		*Buffer = 0;

-		DPRINT("KeyName: <%s>\n", Buffer);
+		DPRINT("KeyName: <%S>\n", Buffer);

if (Delete)
    	{
@@ -424,17 +424,17 @@

if (Delete || (Flags & FLG_ADDREG_OVERWRITEONLY))
    	{
-			if (RegOpenKeyA (NULL, Buffer, &KeyHandle) != ERROR_SUCCESS)
+			if (RegOpenKeyW (NULL, Buffer, &KeyHandle) != ERROR_SUCCESS)
    		{
-				DPRINT("RegOpenKey(%s) failed\n", Buffer);
+				DPRINT("RegOpenKey(%S) failed\n", Buffer);
    			continue;  /* ignore if it doesn't exist */
    		}
    	}
    	else
    	{
-			if (RegCreateKeyA (NULL, Buffer, &KeyHandle) != ERROR_SUCCESS)
+			if (RegCreateKeyW (NULL, Buffer, &KeyHandle) != ERROR_SUCCESS)
    		{
-				DPRINT("RegCreateKey(%s) failed\n", Buffer);
+				DPRINT("RegCreateKey(%S) failed\n", Buffer);
    			continue;
    		}
    	}
@@ -475,12 +475,12 @@
    	return FALSE;
    }

-	if (!registry_callback (hInf, "DelReg", TRUE))
+	if (!registry_callback (hInf, L"DelReg", TRUE))
    {
    	DPRINT1 ("registry_callback() for DelReg failed\n");
    }

-	if (!registry_callback (hInf, "AddReg", FALSE))
+	if (!registry_callback (hInf, L"AddReg", FALSE))
    {
    	DPRINT1 ("registry_callback() for AddReg failed\n");
    }
Index: tools/mkhive/registry.c
===================================================================
--- tools/mkhive/registry.c	(revision 46987)
+++ tools/mkhive/registry.c	(working copy)
@@ -25,9 +25,9 @@

/*
  * TODO:
- *	- Implement RegDeleteKey()
+ *	- Implement RegDeleteKeyW()
  *	- Implement RegEnumValue()
- *	- Implement RegQueryValueExA()
+ *	- Implement RegQueryValueExW()
  */

#include <stdlib.h>
@@ -246,18 +246,37 @@
 }

LONG WINAPI
-RegDeleteKeyA(HKEY Key,
-	     LPCSTR Name)
+RegDeleteKeyW(IN HKEY hKey,
+	      IN LPCWSTR lpSubKey)
 {
-  if (Name != NULL && strchr(Name, '\') != NULL)
-    return(ERROR_INVALID_PARAMETER);
+	if (lpSubKey != NULL && wcschr(lpSubKey, L'\') != NULL)
+		return(ERROR_INVALID_PARAMETER);

-  DPRINT1("FIXME!\n");
+	DPRINT1("RegDeleteKeyW: FIXME!\n");

-  return(ERROR_SUCCESS);
+	return(ERROR_SUCCESS);
 }

LONG WINAPI
+RegDeleteKeyA(IN HKEY hKey,
+	      IN LPCSTR lpSubKey)
+{
+	PWSTR lpSubKeyW;
+	LONG rc;
+
+	if (lpSubKey != NULL && strchr(lpSubKey, '\') != NULL)
+		return(ERROR_INVALID_PARAMETER);
+
+	lpSubKeyW = MultiByteToWideChar(lpSubKey);
+	if (!lpSubKeyW)
+		return ERROR_OUTOFMEMORY;
+
+	rc = RegDeleteKeyW(hKey, lpSubKeyW);
+	free(lpSubKeyW);
+	return rc;
+}
+
+LONG WINAPI
 RegOpenKeyW(
    IN HKEY hKey,
    IN LPCWSTR lpSubKey,

Index: lib/lib.mak
===================================================================
--- lib/lib.mak	(revision 46987)
+++ lib/lib.mak	(working copy)
@@ -1,7 +1,7 @@
 LIB_BASE = lib
 LIB_BASE_ = $(LIB_BASE)$(SEP)

-include lib/inflib/inflib.mak
+include lib/newinflib/inflib.mak
 ifeq ($(ARCH),powerpc)
 include lib/ppcmmu/ppcmmu.mak
 endif

[Sylvain Petreolle]

Are the usetup changes implying a unicode conversion of reactos.dff ?
Doing tests and looking at testbot output shows that usetup is having problem parsing the file.

(base/setup/usetup/interface/usetup.c:2758) SetupFindFirstLine() failed (1st stage)It also fails finding gecko cab file in 2nd stage.

Kind regards,
Sylvain Petreolle

----- Message d'origine ----
> De : Eric Kohl [e-mail]
> À : ReactOS Development List [e-mail]
> Envoyé le : Ven 23 avril 2010, 11 h 42 min 41 s
> Objet : [ros-dev] I need help fixing rbuild and testing builds on *nix machines!
>
> Hi!

In order to fix bug #2482 I needed to make inflib Unicode-aware. My
> local ReactOS setup is curently able to build Boot-CD and Live-CD using Unicode 
> hive*.inf files.

Rbuild is the only tool that still uses the original
> inflib. Could someone who knows more about C++ and STL than I do have a look at 
> rbuild and make it work with the new inflib?

I can provide a patch that
> includes newinflib (Unicode-aware inflib) and  my changes to usetup and 
> mkhive.

I also need someone who can test the whole patch on a *nix
> machine. Any 
> volunters?

Regards,
Eric

_______________________________________________
Ros-dev
> mailing list
> href="mailto:[e-mail]">[e-mail]
> href="http://www.reactos.org/mailman/listinfo/ros-dev" target=_blank
> > http://www.reactos.org/mailman/listinfo/ros-dev

[Eric Kohl]

Hello Sylvain!

The goal is to convert most of the *.inf files to Unicode. The 
conversion is optional, not mandatory. In cases where localized strings 
should be use a conversion is needed. For example syssetup.inf could 
keep the names of all start menu folders and items for all supported 
languages in localized "Strings" sections (eg "[Strings.0407]"). Once 
you install ReactOS the start menu will be localized correctly, no more 
"Accessories" folder for french, spanish, greek or turkish users.

The Unicode-enabled inf parser has at least one or two major bugs that 
cause a heap space corruption on linux machines. It should be either an 
off-by-one bug or a missing "* sizeof(WCHAR)" somewhere because the 
ASCII/ANSI version worked quite well for a few years.

Regards
Eric

Sylvain Petreolle schrieb:
> Are the usetup changes implying a unicode conversion of reactos.dff ?
> Doing tests and looking at testbot output shows that usetup is having problem parsing the file.
>
> (base/setup/usetup/interface/usetup.c:2758) SetupFindFirstLine() failed (1st stage)It also fails finding gecko cab file in 2nd stage.
>
> Kind regards,
> Sylvain Petreolle
>
> ----- Message d'origine ----
> > De : Eric Kohl [e-mail]
> > À : ReactOS Development List [e-mail]
> > Envoyé le : Ven 23 avril 2010, 11 h 42 min 41 s
> > Objet : [ros-dev] I need help fixing rbuild and testing builds on *nix machines!
> >
> > Hi!
>
> In order to fix bug #2482 I needed to make inflib Unicode-aware. My
> > local ReactOS setup is curently able to build Boot-CD and Live-CD using Unicode 
> > hive*.inf files.
>
> Rbuild is the only tool that still uses the original
> > inflib. Could someone who knows more about C++ and STL than I do have a look at 
> > rbuild and make it work with the new inflib?
>
> I can provide a patch that
> > includes newinflib (Unicode-aware inflib) and  my changes to usetup and 
> > mkhive.
>
> I also need someone who can test the whole patch on a *nix
> > machine. Any 
> > volunters?
>
> Regards,
> Eric
>
> Ros-dev
> > mailing list
> > href="mailto:[e-mail]">[e-mail]
> > href="http://www.reactos.org/mailman/listinfo/ros-dev" target=_blank
> > > http://www.reactos.org/mailman/listinfo/ros-dev
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev