How does one make a MP build anyway?

[WaxDragon]

Gang,

   I have been unsuccessful in building a working MP bootcd for a
while now.  I conferred with Hartmut about some of the details, but
still cannot get it working.

This is what I tried:

1. Set MP=1 in config.xml
2. Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut

Index: halx86/up/halup.xml
===================================================================
--- halx86/up/halup.xml (revision 20282)
+++ halx86/up/halup.xml (working copy)
@@ -1,6 +1,6 @@
 <module name="halup" type="kernelmodedll">
        <importlibrary definition="../../hal/hal.def" />
-       <bootstrap base="reactos" nameoncd="hal.dll" />
+       <bootstrap base="reactos" />
        <include base="hal_generic">../include</include>
        <include base="ntoskrnl">include</include>
        <define name="_DISABLE_TIDENTS" />
Index: halx86/mp/halmp.xml
===================================================================
--- halx86/mp/halmp.xml (revision 20282)
+++ halx86/mp/halmp.xml (working copy)
@@ -1,6 +1,6 @@
 <module name="halmp" type="kernelmodedll">
        <importlibrary definition="../../hal/hal.def" />
-       <bootstrap base="reactos" />
+       <bootstrap base="reactos" nameoncd="hal.dll"/>
        <include base="hal_generic">../include</include>
        <include base="ntoskrnl">include</include>
        <define name="_DISABLE_TIDENTS" />

3. make bootcd, as normal

The resulting bootcd fails to boot under "qemu -smp 2" with the following
trace:

(./ntoskrnl/ke/main.c:295)
---------------------------------------------------------------
(./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
(hal/halx86/mp/apic.c:420) Getting VERSION: 50011
(hal/halx86/mp/apic.c:423) Getting VERSION: 50011
(hal/halx86/mp/apic.c:454) Getting ID: 0
(hal/halx86/mp/apic.c:457) Getting ID: f000000
(hal/halx86/mp/apic.c:554) CPU0:
(hal/halx86/mp/apic.c:555)   Physical APIC id: 0
(hal/halx86/mp/apic.c:556)   Logical APIC id:  0
(hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
(hal/halx86/mp/apic.c:584) CPU0:
(hal/halx86/mp/apic.c:585)   Physical APIC id: 0
(hal/halx86/mp/apic.c:586)   Logical APIC id:  1
(hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
(hal/halx86/mp/apic.c:588) 0
(hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
Used memory 131072Kb
(./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
0x8013c000, Init = 0x80139000
(./ntoskrnl/mm/mm.c:283) No current process
(hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
(hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
(hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
(hal/halx86/mp/apic.c:554) CPU1:
(hal/halx86/mp/apic.c:555)   Physical APIC id: 1
(hal/halx86/mp/apic.c:556)   Logical APIC id:  0
(hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
(hal/halx86/mp/apic.c:584) CPU1:
(hal/halx86/mp/apic.c:585)   Physical APIC id: 1
(hal/halx86/mp/apic.c:586)   Logical APIC id:  2
(hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
(hal/halx86/mp/apic.c:588) 1
(hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
CPU 1 is now running
(ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
(./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
the ipi routine
KeBugCheck at ./ntoskrnl/ke/ipi.c:89
A problem has been detected and ReactOS has been shut down to prevent
damage to your computer.

Technical information:

*** STOP: 0x00000000 (0x00000000,0x00000000,0x00000000,0x00000000)

Frames:
<ntoskrnl.exe:2630>    ./ntoskrnl/ke/bug.c:0 ()
<ntoskrnl.exe:2668>    ./ntoskrnl/ke/bug.c:504 (KeBugCheck)
<ntoskrnl.exe:414f>     ./ntoskrnl/ke/ipi.c:0 ()
<ntoskrnl.exe:4470>    ./ntoskrnl/ke/ipi.c:0 ()
<ntoskrnl.exe:44df>     ./ntoskrnl/ke/ipi.c:174 (KeIpiGenericCall)
<ntoskrnl.exe:abd71>  ./ntoskrnl/mm/i386/page.c:1043 (MmDeleteVirtualMapping)
<ntoskrnl.exe:ac891>  ./ntoskrnl/mm/i386/page.c:431 (MmDeletePageTable)
<ntoskrnl.exe:d52ce>  ./ntoskrnl/mm/mminit.c:459 (MmInit3)
<ntoskrnl.exe:cf7c0>   ./ntoskrnl/ex/init.c:622 (ExpInitializeExecutive)
<ntoskrnl.exe:66fc>    ./ntoskrnl/ke/main.c:106 (KiSystemStartup)
<ntoskrnl.exe:cd851> ./ntoskrnl/ke/main.c:300 (_main)
<ntoskrnl.exe:104b>   ./ntoskrnl/ke/i386/main_asm.S:46 (NtProcessStartup)


An MP bootcd I made several months ago (~18xxx) did the same thing on
a Abit BP6 with dual Celerons, so I'm fairly confident qemu is acting
properly.  I still have the BP6 and am willing to test MP builds, but
I wasted about 10 CD-R's trying to get a working MP bootcd before
giving up.

Am I just doing something wrong?  I would like to get this worked out
and documented somewhere.

WD
--
<Russell> argh
<Russell> iterator shenanigans :/

[Hartmut Birr]

WaxDragon wrote:
>   Gang,
>
>     I have been unsuccessful in building a working MP bootcd for a
>  while now.  I conferred with Hartmut about some of the details, but
>  still cannot get it working.
>
>  This is what I tried:
>
>  1. Set MP=1 in config.xml
>  2. Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut
>
>  Index: halx86/up/halup.xml
>  ===================================================================
>  --- halx86/up/halup.xml (revision 20282)
>  +++ halx86/up/halup.xml (working copy)
>  @@ -1,6 +1,6 @@
>   <module name="halup" type="kernelmodedll">
>          <importlibrary definition="../../hal/hal.def" />
>  -       <bootstrap base="reactos" nameoncd="hal.dll" />
>  +       <bootstrap base="reactos" />
>          <include base="hal_generic">../include</include>
>          <include base="ntoskrnl">include</include>
>          <define name="_DISABLE_TIDENTS" />
>  Index: halx86/mp/halmp.xml
>  ===================================================================
>  --- halx86/mp/halmp.xml (revision 20282)
>  +++ halx86/mp/halmp.xml (working copy)
>  @@ -1,6 +1,6 @@
>   <module name="halmp" type="kernelmodedll">
>          <importlibrary definition="../../hal/hal.def" />
>  -       <bootstrap base="reactos" />
>  +       <bootstrap base="reactos" nameoncd="hal.dll"/>
>          <include base="hal_generic">../include</include>
>          <include base="ntoskrnl">include</include>
>          <define name="_DISABLE_TIDENTS" />
>
>  3. make bootcd, as normal
>
>     
I can build a smp boot cd. The smp build runs on my smp machine (ASUS
P2B-DS motherboard with 2xPIII 500MHz). I've also boot the smp build on
my amd64. I've update my pc with a amd64x2 cpu. Since the update, the
smp build won't boot. Ros hangs before I can see any debug messages.

>   The resulting bootcd fails to boot under "qemu
> -smp 2" with the following trace:
>
>  (./ntoskrnl/ke/main.c:295)
>  ---------------------------------------------------------------
>  (./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
>  (hal/halx86/mp/apic.c:420) Getting VERSION: 50011
>  (hal/halx86/mp/apic.c:423) Getting VERSION: 50011
>  (hal/halx86/mp/apic.c:454) Getting ID: 0
>  (hal/halx86/mp/apic.c:457) Getting ID: f000000
>  (hal/halx86/mp/apic.c:554) CPU0:
>  (hal/halx86/mp/apic.c:555)   Physical APIC id: 0
>  (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
>  (hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
>  (hal/halx86/mp/apic.c:584) CPU0:
>  (hal/halx86/mp/apic.c:585)   Physical APIC id: 0
>  (hal/halx86/mp/apic.c:586)   Logical APIC id:  1
>  (hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
>  (hal/halx86/mp/apic.c:588) 0
>  (hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
>  Used memory 131072Kb
>  (./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
>  0x8013c000, Init = 0x80139000
>  (./ntoskrnl/mm/mm.c:283) No current process
>  (hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
>  (hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
>  (hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
>  (hal/halx86/mp/apic.c:554) CPU1:
>  (hal/halx86/mp/apic.c:555)   Physical APIC id: 1
>  (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
>  (hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
>  (hal/halx86/mp/apic.c:584) CPU1:
>  (hal/halx86/mp/apic.c:585)   Physical APIC id: 1
>  (hal/halx86/mp/apic.c:586)   Logical APIC id:  2
>  (hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
>  (hal/halx86/mp/apic.c:588) 1
>  (hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
>  CPU 1 is now running
>  (ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
>  (./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
>  the ipi routine 
The ipi interrupt isn't sent to the second cpu. I've looked
to the qemu
apic code. The 'fixed delivery mode' isn't implemented in
apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
linux or ReactOS will not run with qemu.

- Hartmut

[Filip Navara]

Hartmut Birr wrote:

>  The ipi interrupt isn't sent to the second cpu.
> I've looked to the qemu
> apic code. The 'fixed delivery mode' isn't implemented in
> apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
> linux or ReactOS will not run with qemu.
>   
>  
It is implemented, the single "break;" statement means that just no 
additional processing is required/implemented.

- Filip

[Hartmut Birr]

Filip Navara wrote:
>   It is implemented, the single "break;"
> statement means that just no
>  additional processing is required/implemented. 
Your are right. 
It seems, that the current crash is triggered by APICSyncArbIDs. Qemu
reinitialize the local apic at this point. The logical apic ids are
reset. I can change qemu or reactos a little bit to fix this. Reactos
runs a little bit longer in qemu. The change has no effect on my smp
machine.

- Hartmut

M:\Sandbox\ros_work\reactos>svn diff hal\halx86\mp\apic.c
Index: hal/halx86/mp/apic.c
===================================================================
--- hal/halx86/mp/apic.c        (Revision 20275)
+++ hal/halx86/mp/apic.c        (Arbeitskopie)
@@ -691,7 +691,8 @@
    }

    DPRINT("Synchronizing Arb IDs.\n");
-   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_ICR0_LEVEL |
APIC_DM_INIT);
+//   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_ICR0_LEVEL |
APIC_DM_INIT);
+   APICWrite(APIC_ICR0, APIC_ICR0_DESTS_ALL | APIC_DM_INIT | APIC_ICR0_TM);
 }
 #endif

[WaxDragon]

I use the damnsmalllinux (DSL) livecd, and it works just fine with
-smp 2 and 4 under qemu 0.8.0, and as I said in the original post, I
see the same error on my Abit BP6.

The first question I want answered is "Am I building the MP build correctly?"

On 12/21/05, Hartmut Birr <osexpert(a)googlemail.com> wrote:
>   WaxDragon wrote:
> >   Gang,
> >
> >     I have been unsuccessful in building a working MP bootcd for a
> >  while now.  I conferred with Hartmut about some of the details, but
> >  still cannot get it working.
> >
> >  This is what I tried:
> >
> >  1. Set MP=1 in config.xml
> >  2. Hacked the hal xml files to make halmp.dll hal.dll on the bootcd, per Harmut
> >
> >  Index: halx86/up/halup.xml
> >  ===================================================================
> >  --- halx86/up/halup.xml (revision 20282)
> >  +++ halx86/up/halup.xml (working copy)
> >  @@ -1,6 +1,6 @@
> >   <module name="halup" type="kernelmodedll">
> >          <importlibrary definition="../../hal/hal.def" />
> >  -       <bootstrap base="reactos" nameoncd="hal.dll" />
> >  +       <bootstrap base="reactos" />
> >          <include base="hal_generic">../include</include>
> >          <include base="ntoskrnl">include</include>
> >          <define name="_DISABLE_TIDENTS" />
> >  Index: halx86/mp/halmp.xml
> >  ===================================================================
> >  --- halx86/mp/halmp.xml (revision 20282)
> >  +++ halx86/mp/halmp.xml (working copy)
> >  @@ -1,6 +1,6 @@
> >   <module name="halmp" type="kernelmodedll">
> >          <importlibrary definition="../../hal/hal.def" />
> >  -       <bootstrap base="reactos" />
> >  +       <bootstrap base="reactos" nameoncd="hal.dll"/>
> >          <include base="hal_generic">../include</include>
> >          <include base="ntoskrnl">include</include>
> >          <define name="_DISABLE_TIDENTS" />
> >
> >  3. make bootcd, as normal
> >
> >  
>  I can build a smp boot cd. The smp build runs on my smp machine (ASUS
>  P2B-DS motherboard with 2xPIII 500MHz). I've also boot the smp build on
>  my amd64. I've update my pc with a amd64x2 cpu. Since the update, the
>  smp build won't boot. Ros hangs before I can see any debug messages.
>
> >   The resulting bootcd fails to boot under
> > "qemu -smp 2" with the following trace:
> >
> >  (./ntoskrnl/ke/main.c:295)
> >  ---------------------------------------------------------------
> >  (./ntoskrnl/ke/main.c:296) ReactOS 0.3-SVN (Build 20051221-r20282)
> >  (hal/halx86/mp/apic.c:420) Getting VERSION: 50011
> >  (hal/halx86/mp/apic.c:423) Getting VERSION: 50011
> >  (hal/halx86/mp/apic.c:454) Getting ID: 0
> >  (hal/halx86/mp/apic.c:457) Getting ID: f000000
> >  (hal/halx86/mp/apic.c:554) CPU0:
> >  (hal/halx86/mp/apic.c:555)   Physical APIC id: 0
> >  (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> >  (hal/halx86/mp/apic.c:557) 00000000 00000000 f0000000
> >  (hal/halx86/mp/apic.c:584) CPU0:
> >  (hal/halx86/mp/apic.c:585)   Physical APIC id: 0
> >  (hal/halx86/mp/apic.c:586)   Logical APIC id:  1
> >  (hal/halx86/mp/apic.c:587) 00000000 01000000 f0000000
> >  (hal/halx86/mp/apic.c:588) 0
> >  (hal/halx86/mp/apic.c:619) enabled ExtINT on CPU#0
> >  Used memory 131072Kb
> >  (./ntoskrnl/mm/mminit.c:386) Kernel Stack Limits. InitTop =
> >  0x8013c000, Init = 0x80139000
> >  (./ntoskrnl/mm/mm.c:283) No current process
> >  (hal/halx86/mp/processor_mp.c:112) Attempting to boot CPU 1
> >  (hal/halx86/mp/apic.c:1014) Attempting to boot CPU 1
> >  (hal/halx86/mp/apic.c:1044) 80572be0 25000 80001000 0
> >  (hal/halx86/mp/apic.c:554) CPU1:
> >  (hal/halx86/mp/apic.c:555)   Physical APIC id: 1
> >  (hal/halx86/mp/apic.c:556)   Logical APIC id:  0
> >  (hal/halx86/mp/apic.c:557) 01000000 00000000 f0000000
> >  (hal/halx86/mp/apic.c:584) CPU1:
> >  (hal/halx86/mp/apic.c:585)   Physical APIC id: 1
> >  (hal/halx86/mp/apic.c:586)   Logical APIC id:  2
> >  (hal/halx86/mp/apic.c:587) 01000000 02000000 f0000000
> >  (hal/halx86/mp/apic.c:588) 1
> >  (hal/halx86/mp/apic.c:624) masked ExtINT on CPU#1
> >  CPU 1 is now running
> >  (ntoskrnl/ke/i386/exp.c:539) Ignoring P6 Local APIC Spurious Interrupt Bug...
> >  (./ntoskrnl/ke/ipi.c:88) CPU0, waiting longer than 5 seconds to start
> >  the ipi routine 
>  The ipi interrupt isn't sent to the second cpu. I've
> looked to the qemu
>  apic code. The 'fixed delivery mode' isn't implemented in
>  apic_bus_deliver (hw\apic.c). This means,  a smp version of windows,
>  linux or ReactOS will not run with qemu.
>
>  - Hartmut
>  _______________________________________________
>  Ros-dev mailing list
>  Ros-dev(a)reactos.org
>  http://www.reactos.org/mailman/listinfo/ros-dev
>  

--
<Russell> argh
<Russell> iterator shenanigans :/