How does one make a MP build anyway?

[WaxDragon]

Gang,

I have been unsuccessful in building a working MP bootcd for a
while now.  I conferred with Hartmut about some of the details, but
still cannot get it working.

This is what I tried:

1. Set MP=1 in config.xml
2. Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut

Index: halx86/up/halup.xml
===================================================================
--- halx86/up/halup.xml (revision 20282)
+++ halx86/up/halup.xml (working copy)
@@ -1,6 +1,6 @@
 <module name="halup" type="kernelmodedll">
        <importlibrary definition="../../hal/hal.def" />
-       <bootstrap base="reactos" nameoncd="hal.dll" />
+       <bootstrap base="reactos" />
        <include base="hal_generic">../include</include>
        <include base="ntoskrnl">include</include>
        <define name="_DISABLE_TIDENTS" />
Index: halx86/mp/halmp.xml
===================================================================
--- halx86/mp/halmp.xml (revision 20282)
+++ halx86/mp/halmp.xml (working copy)
@@ -1,6 +1,6 @@
 <module name="halmp" type="kernelmodedll">
        <importlibrary definition="../../hal/hal.def" />
-       <bootstrap base="reactos" />
+       <bootstrap base="reactos" nameoncd="hal.dll"/>
        <include base="hal_generic">../include</include>
        <include base="ntoskrnl">include</include>
        <define name="_DISABLE_TIDENTS" />

3. make bootcd, as normal

The resulting bootcd fails to boot under "qemu -smp 2" with the following trace:

(./ntoskrnl/ke/main.c:295)
---------------------------------------------------------------
(./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
(hal/halx86/mp/apic.c:420) Getting VERSION: 50011
(hal/halx86/mp/apic.c:423) Getting VERSION: 50011
(hal/halx86/mp/apic.c:454) Getting ID: 0
(hal/halx86/mp/apic.c:457) Getting ID: f000000
(hal/halx86/mp/apic.c:554) CPU0:
(hal/halx86/mp/apic.c:555)   Physical APIC id: 0
(hal/halx86/mp/apic.c:556)   Logical APIC id:  0
(hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
(hal/halx86/mp/apic.c:584) CPU0:
(hal/halx86/mp/apic.c:585)   Physical APIC id: 0
(hal/halx86/mp/apic.c:586)   Logical APIC id:  1
(hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
(hal/halx86/mp/apic.c:588) 0
(hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
Used memory 131072Kb
(./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
0x8013c000, Init = 0x80139000
(./ntoskrnl/mm/mm.c:283) No current process
(hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
(hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
(hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
(hal/halx86/mp/apic.c:554) CPU1:
(hal/halx86/mp/apic.c:555)   Physical APIC id: 1
(hal/halx86/mp/apic.c:556)   Logical APIC id:  0
(hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
(hal/halx86/mp/apic.c:584) CPU1:
(hal/halx86/mp/apic.c:585)   Physical APIC id: 1
(hal/halx86/mp/apic.c:586)   Logical APIC id:  2
(hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
(hal/halx86/mp/apic.c:588) 1
(hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
CPU 1 is now running
(ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
(./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
the ipi routine
KeBugCheck at ./ntoskrnl/ke/ipi.c:89
A problem has been detected and ReactOS has been shut down to prevent
damage to your computer.

Technical information:

*** STOP: 0x00000000 (0x00000000,0x00000000,0x00000000,0x00000000)

Frames:
ntoskrnl.exe:2630    ./ntoskrnl/ke/bug.c:0 ()
ntoskrnl.exe:2668    ./ntoskrnl/ke/bug.c:504 (KeBugCheck)
ntoskrnl.exe:414f     ./ntoskrnl/ke/ipi.c:0 ()
ntoskrnl.exe:4470    ./ntoskrnl/ke/ipi.c:0 ()
ntoskrnl.exe:44df     ./ntoskrnl/ke/ipi.c:174 (KeIpiGenericCall)
ntoskrnl.exe:abd71  ./ntoskrnl/mm/i386/page.c:1043 (MmDeleteVirtualMapping)
ntoskrnl.exe:ac891  ./ntoskrnl/mm/i386/page.c:431 (MmDeletePageTable)
ntoskrnl.exe:d52ce  ./ntoskrnl/mm/mminit.c:459 (MmInit3)
ntoskrnl.exe:cf7c0   ./ntoskrnl/ex/init.c:622 (ExpInitializeExecutive)
ntoskrnl.exe:66fc    ./ntoskrnl/ke/main.c:106 (KiSystemStartup)
ntoskrnl.exe:cd851 ./ntoskrnl/ke/main.c:300 (_main)
ntoskrnl.exe:104b   ./ntoskrnl/ke/i386/main_asm.S:46 (NtProcessStartup)

An MP bootcd I made several months ago (~18xxx) did the same thing on
a Abit BP6 with dual Celerons, so I'm fairly confident qemu is acting
properly.  I still have the BP6 and am willing to test MP builds, but
I wasted about 10 CD-R's trying to get a working MP bootcd before
giving up.

Am I just doing something wrong?  I would like to get this worked out
and documented somewhere.

WD
--
<Russell> argh
<Russell> iterator shenanigans :/

[Hartmut Birr]

WaxDragon wrote:
> Gang,
>
> I have been unsuccessful in building a working MP bootcd for a
> while now.  I conferred with Hartmut about some of the details, but
> still cannot get it working.
>
> This is what I tried:
> Set MP=1 in config.xml
> Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut
> Index: halx86/up/halup.xml
>
> --- halx86/up/halup.xml (revision 20282)
> +++ halx86/up/halup.xml (working copy)
> @@ -1,6 +1,6 @@
>
>  <module name="halup" type="kernelmodedll">
>         <importlibrary definition="../../hal/hal.def" />
> -       <bootstrap base="reactos" nameoncd="hal.dll" />
> +       <bootstrap base="reactos" />
>         <include base="hal_generic">../include</include>
>         <include base="ntoskrnl">include</include>
>         <define name="_DISABLE_TIDENTS" />
> Index: halx86/mp/halmp.xml
> ===================================================================
> --- halx86/mp/halmp.xml (revision 20282)
> +++ halx86/mp/halmp.xml (working copy)
> @@ -1,6 +1,6 @@
>  <module name="halmp" type="kernelmodedll">
>         <importlibrary definition="../../hal/hal.def" />
> -       <bootstrap base="reactos" />
> +       <bootstrap base="reactos" nameoncd="hal.dll"/>
>         <include base="hal_generic">../include</include>
>         <include base="ntoskrnl">include</include>
>         <define name="_DISABLE_TIDENTS" />
> make bootcd, as normal

I can build a smp boot cd. The smp build runs on my smp machine (ASUS
P2B-DS motherboard with 2xPIII 500MHz). I've also boot the smp build on
my amd64. I've update my pc with a amd64x2 cpu. Since the update, the
smp build won't boot. Ros hangs before I can see any debug messages.
> The resulting bootcd fails to boot under "qemu -smp 2" with the following trace:
> (./ntoskrnl/ke/main.c:295)
>
> (./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
> (hal/halx86/mp/apic.c:420) Getting VERSION: 50011
> (hal/halx86/mp/apic.c:423) Getting VERSION: 50011
> (hal/halx86/mp/apic.c:454) Getting ID: 0
> (hal/halx86/mp/apic.c:457) Getting ID: f000000
> (hal/halx86/mp/apic.c:554) CPU0:
> (hal/halx86/mp/apic.c:555)   Physical APIC id: 0
> (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> (hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
> (hal/halx86/mp/apic.c:584) CPU0:
> (hal/halx86/mp/apic.c:585)   Physical APIC id: 0
> (hal/halx86/mp/apic.c:586)   Logical APIC id:  1
> (hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
> (hal/halx86/mp/apic.c:588) 0
> (hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
> Used memory 131072Kb
> (./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
> 0x8013c000, Init = 0x80139000
> (./ntoskrnl/mm/mm.c:283) No current process
> (hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
> (hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
> (hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
> (hal/halx86/mp/apic.c:554) CPU1:
> (hal/halx86/mp/apic.c:555)   Physical APIC id: 1
> (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> (hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
> (hal/halx86/mp/apic.c:584) CPU1:
> (hal/halx86/mp/apic.c:585)   Physical APIC id: 1
> (hal/halx86/mp/apic.c:586)   Logical APIC id:  2
> (hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
> (hal/halx86/mp/apic.c:588) 1
> (hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
> CPU 1 is now running
> (ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
> (./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
> the ipi routine

The ipi interrupt isn't sent to the second cpu. I've looked to the qemu
apic code. The 'fixed delivery mode' isn't implemented in
apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
linux or ReactOS will not run with qemu.

- Hartmut

[Filip Navara]

Hartmut Birr wrote:
> The ipi interrupt isn't sent to the second cpu. I've looked to the qemu
> apic code. The 'fixed delivery mode' isn't implemented in
> apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
> linux or ReactOS will not run with qemu.

It is implemented, the single "break;" statement means that just no 
additional processing is required/implemented.

- Filip

[Hartmut Birr]

Filip Navara wrote:
> It is implemented, the single "break;" statement means that just no
> additional processing is required/implemented.

Your are right. 
It seems, that the current crash is triggered by APICSyncArbIDs. Qemu
reinitialize the local apic at this point. The logical apic ids are
reset. I can change qemu or reactos a little bit to fix this. Reactos
runs a little bit longer in qemu. The change has no effect on my smp
machine.

- Hartmut

M:\Sandbox\ros_work\reactos>svn diff hal\halx86\mp\apic.c
Index: hal/halx86/mp/apic.c
===================================================================
--- hal/halx86/mp/apic.c        (Revision 20275)
+++ hal/halx86/mp/apic.c        (Arbeitskopie)
@@ -691,7 +691,8 @@
    }

DPRINT("Synchronizing Arb IDs.\n");
-   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_ICR0_LEVEL |
APIC_DM_INIT);
+//   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_ICR0_LEVEL |
APIC_DM_INIT);
+   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_DM_INIT | APIC_ICR0_TM);
 }
 #endif

[WaxDragon]

I use the damnsmalllinux (DSL) livecd, and it works just fine with
-smp 2 and 4 under qemu 0.8.0, and as I said in the original post, I
see the same error on my Abit BP6.

The first question I want answered is "Am I building the MP build correctly?"

On 12/21/05, Hartmut Birr [e-mail] wrote:
> WaxDragon wrote:
> > Gang,
> >
> > I have been unsuccessful in building a working MP bootcd for a
> > while now.  I conferred with Hartmut about some of the details, but
> > still cannot get it working.
> >
> > This is what I tried:
> > Set MP=1 in config.xml
> > Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut
> > Index: halx86/up/halup.xml
> >
> > --- halx86/up/halup.xml (revision 20282)
> > +++ halx86/up/halup.xml (working copy)
> > @@ -1,6 +1,6 @@
> >
> >  <module name="halup" type="kernelmodedll">
> >         <importlibrary definition="../../hal/hal.def" />
> > -       <bootstrap base="reactos" nameoncd="hal.dll" />
> > +       <bootstrap base="reactos" />
> >         <include base="hal_generic">../include</include>
> >         <include base="ntoskrnl">include</include>
> >         <define name="_DISABLE_TIDENTS" />
> > Index: halx86/mp/halmp.xml
> > ===================================================================
> > --- halx86/mp/halmp.xml (revision 20282)
> > +++ halx86/mp/halmp.xml (working copy)
> > @@ -1,6 +1,6 @@
> >  <module name="halmp" type="kernelmodedll">
> >         <importlibrary definition="../../hal/hal.def" />
> > -       <bootstrap base="reactos" />
> > +       <bootstrap base="reactos" nameoncd="hal.dll"/>
> >         <include base="hal_generic">../include</include>
> >         <include base="ntoskrnl">include</include>
> >         <define name="_DISABLE_TIDENTS" />
> > make bootcd, as normal
>
> I can build a smp boot cd. The smp build runs on my smp machine (ASUS
> P2B-DS motherboard with 2xPIII 500MHz). I've also boot the smp build on
> my amd64. I've update my pc with a amd64x2 cpu. Since the update, the
> smp build won't boot. Ros hangs before I can see any debug messages.
> > The resulting bootcd fails to boot under "qemu -smp 2" with the following trace:
> > (./ntoskrnl/ke/main.c:295)
> >
> > (./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
> > (hal/halx86/mp/apic.c:420) Getting VERSION: 50011
> > (hal/halx86/mp/apic.c:423) Getting VERSION: 50011
> > (hal/halx86/mp/apic.c:454) Getting ID: 0
> > (hal/halx86/mp/apic.c:457) Getting ID: f000000
> > (hal/halx86/mp/apic.c:554) CPU0:
> > (hal/halx86/mp/apic.c:555)   Physical APIC id: 0
> > (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> > (hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
> > (hal/halx86/mp/apic.c:584) CPU0:
> > (hal/halx86/mp/apic.c:585)   Physical APIC id: 0
> > (hal/halx86/mp/apic.c:586)   Logical APIC id:  1
> > (hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
> > (hal/halx86/mp/apic.c:588) 0
> > (hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
> > Used memory 131072Kb
> > (./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
> > 0x8013c000, Init = 0x80139000
> > (./ntoskrnl/mm/mm.c:283) No current process
> > (hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
> > (hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
> > (hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
> > (hal/halx86/mp/apic.c:554) CPU1:
> > (hal/halx86/mp/apic.c:555)   Physical APIC id: 1
> > (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> > (hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
> > (hal/halx86/mp/apic.c:584) CPU1:
> > (hal/halx86/mp/apic.c:585)   Physical APIC id: 1
> > (hal/halx86/mp/apic.c:586)   Logical APIC id:  2
> > (hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
> > (hal/halx86/mp/apic.c:588) 1
> > (hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
> > CPU 1 is now running
> > (ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
> > (./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
> > the ipi routine
>
> The ipi interrupt isn't sent to the second cpu. I've looked to the qemu
> apic code. The 'fixed delivery mode' isn't implemented in
> apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
> linux or ReactOS will not run with qemu.
> Hartmut
>
> Ros-dev mailing list
> [e-mail]
> http://www.reactos.org/mailman/listinfo/ros-dev

--
<Russell> argh
<Russell> iterator shenanigans :/