Multicore

FlintOS runs one scheduler across every core. A task is a task; the core it lands on is a scheduling decision, not a property of the code.

That is the whole design, and it only works for symmetric cores — identical instruction set, identical memory view, identical clock. Asymmetric packages (a Cortex-M7 next to an M0) are deliberately out of scope. See Asymmetric cores for why.

On the ESP32 that means the PRO CPU and the APP CPU, both Xtensa LX6.

---

Starting the second core

It does not start itself. Something on the first core has to release it:

unsafe {
    arch_xtensa::appcpu::prepare(second_core);
    soc_esp32::appcpu::start(arch_xtensa::appcpu::_flint_appcpu_entry);
}

Then the second core’s entry hands itself to the scheduler and never returns:

#[link_section = ".iram1.second_core"]
extern "C" fn second_core() -> ! {
    unsafe { kernel::boot::join_scheduler() }
}

From that point the two cores are peers. Both take timer ticks, both service interrupts, both run tasks out of the same ready queue.

.iram1 is not optional

The second core has no instruction cache when it starts. Its entry runs from a reset vector, and until the first core calls Cache_Flush(1) and Cache_Read_Enable(1) on its behalf, every byte of flash reads as garbage.

A function in .text lives in flash. Putting the second core’s entry there gets an IllegalInstruction exception, immediately, every time — which reads like a miscompile and is not one.

appcpu::start enables the cache, so anything the entry calls can live in flash. The entry itself cannot.

The three holds

The APP CPU is held by three separate mechanisms, and releasing two of them does nothing observable. All three, in this order:

Hold	Register	Bits
RTC stall	RTC_CNTL_OPTIONS0	[1:0], plus SW_CPU_STALL[25:20]
Clock gate	DPORT_APPCPU_CTRL_B	bit 0
Reset	DPORT_APPCPU_CTRL_A	bit 0

soc_esp32::appcpu::start does all of it. The table is here because a core that stays dark after a partial release gives no diagnostic at all.

---

Where a task runs

By default, anywhere:

Task::new("worker", worker).priority(Priority::Normal(2)).spawn();

Pinned to a core, when it matters:

Task::new("sensor", sensor).priority(Priority::Normal(2)).on_core(1).spawn();

Pinning is a constraint on the scheduler, not a hint. A task pinned to core 1 will sit Ready while core 0 idles rather than run in the wrong place.

When to pin

Pin when the task’s correctness depends on the core, not when you are guessing about performance:

• it touches a peripheral whose interrupt is routed to that core
• it must not compete with an interrupt-heavy core for cycles
• you are measuring one core’s behaviour and need the other one quiet

Otherwise leave it unpinned. The scheduler balances better than a guess does, and a pinned task that did not need pinning is one that cannot use an idle core.

Priority still wins

Affinity narrows the choice; it does not override priority. Each core independently picks the highest-priority task that is Ready, allowed on that core, and not already running elsewhere.

Two consequences worth knowing:

• A high-priority pinned task blocks lower-priority work on its core only.
• The same task never runs on two cores at once, even unpinned — the scheduler checks running_elsewhere before it hands a task out.

---

What is shared, and what is not

Shared, one copy, protected by a lock:

• the scheduler — ready mask, every TCB, the timer list
• the mutex table
• the log buffer and metrics
• DPORT — peripheral clock gating, the interrupt crossbar, the DMA channel selector. Shared hardware rather than kernel state, and unsafe in its own particular way: see Reading DPORT. Go through soc_esp32::dport, never a bare read_volatile.

Per-core, one copy each:

• the current task (current_per_core)
• the pending-switch flag
• the interrupt-nesting depth
• the tick re-arm (each core arms its own comparator)

The tick count advances on the boot core only. Both cores take the interrupt; only one owns the clock. Two cores each incrementing it would make time run at double speed.

---

The lock

One spinlock type, kernel::smp::Spinlock<T>, and it wraps the data rather than sitting beside it:

static SEEN: Spinlock<[[u32; 2]; 3]> = Spinlock::new([[0; 2]; 3]);

SEEN.with(|seen| seen[which][core] += 1);

There is no lock() returning a guard you can hold across arbitrary code. The closure form makes the critical section a lexical region, which is what makes “is this held on the interrupt path” answerable by reading.

Interrupts are masked first

with masks interrupts before it takes the lock, and releases in the opposite order. The other ordering deadlocks: take the lock, then take an interrupt on the same core, then have the handler want the same lock. It never comes back, and it happens under load rather than in testing.

Re-entering panics

Taking a lock you already hold is a bug, not a wait. with panics rather than spinning, because spinning here is a hang with no message — and a hang on core 1 looks exactly like a core that never started.

This is why the panic handler and the log path do not go through the scheduler lock. A panic that blocks on a lock is a panic you never see.

try_with

For the paths that genuinely cannot block:

if let Some(v) = LOCK.try_with(|state| state.snapshot()) {
    // got it
}

Returns None rather than waiting. Used where dropping the update beats stalling — diagnostics, mostly.

---

Getting the other core out of the way

Writing flash means taking SPI1 away from the instruction cache, so for the duration of the operation neither core can fetch from flash. The second one has to be got out of the way, and how is the interesting part.

The obvious way is appcpu::stall() — a hardware freeze. FlintOS used it, and it was safe by an argument about the other side: nothing ran between the stall and the release, so core 1 could be frozen holding a Spinlock and nobody would ask for it. Then IRAM-safe interrupts started running in that window and the argument was gone.

esp-idf (spi_flash_disable_interrupts_caches_and_other_cpu) and NuttX (esp32_spiflash_opstart) both ask instead, and neither freezes:

1. Signal the other core — esp-idf an IPC call, NuttX a semaphore, FlintOS the FROM_CPU_2 cross-core interrupt, routed in the APP crossbar table only so it lands on core 1 and nowhere else.
2. That core disables its own cache, says it is parked, and spins.
3. The initiator waits for “parked”, does the operation, then releases.

Why asking is enough, in this kernel’s terms: Spinlock::with takes the lock inside arch::cs_with, so a lock is only ever held with interrupts masked on the holding core. An interrupt therefore cannot land on core 1 while core 1 holds a lock — so a core parked by an interrupt provably holds none. Same invariant esp-idf relies on, reached from FlintOS’s own primitives.

The stall survives as a bounded fallback, and esp32_flash::PARK_FELL_BACK records it if it ever fires. esp32_flash::PARKS counts the handshakes, so a dead handshake and a working one can be told apart — apps/tests/flashprobe prints both. (apps/tests/smp also starts the second core, to pin tasks across cores; flashprobe is the one that reports the park handshake counters.)

Interrupts during a flash operation

A sector erase is tens of milliseconds. Masking everything for that long is a real-time defect with or without a radio, so FlintOS masks selectively: handlers registered through interrupt::register_iram_safe stay enabled in INTENABLE, everything else is masked, and _flint_trap takes a separate IRAM-only path for the duration — no tick, no scheduler, no timers.

Registering there is two promises, not one:

• IRAM, for the handler and everything it calls. Check the built ELF, not the attributes; #[link_section] says nothing about a copy the optimiser inlined into a caller.
• No locks. See above — this is the register for handlers that run while the other core is parked.

---

Asymmetric cores

Out of scope, on purpose.

Two cores of different architectures usually means two operating systems — the small core running something else entirely, or nothing. Programming against that means picking a memory model, a boot protocol and an IPC mechanism that vary per package, with no way to test the combinations.

Symmetric SMP is a well-defined problem with one answer. Asymmetric is a different design, and starting it before symmetric is solid would produce a worse version of both.

---

Verifying it on hardware

make flash APP=smp BOARD=board-m5-atom-matrix

Three tasks — one pinned to core 0, one to core 1, one unpinned — each recording which core it observed itself on, every iteration. All three sleep, which forces a switch: a pinned task that is never preempted proves nothing.

What a correct run shows:

pin0   core0=N  core1=0
pin1   core0=0  core1=N
float  core0=N  core1=M

Both zeros matter. A pinned task with a nonzero count on the wrong core is a scheduler bug; float with a zero is a second core that never actually ran.

The same app also hammers DPORT from both cores at once, reporting

dport round N: core0 X ops, core1 Y ops, lost 0

lost counts the times a core set its own clock bit and immediately found it gone, which only the other core can have done. It must stay at zero, and both op counts must be nonzero — a lost-update test where one side never ran is a test that passes for the wrong reason.

---

Sources

• ESP32 TRM §31 (DPORT), §29 (RTC_CNTL)
• soc/esp32/src/appcpu.rs — the three holds and the cache enable
• kernel/src/smp.rs — the lock
• kernel/src/scheduler.rs — affinity and per-core current
• soc/esp32/src/dport.rs — the DPORT erratum workaround and its lock
• apps/tests/smp/ — the hardware test