Curriculum and multi-stage training

Krishnapriyan, Gholami, Zhe, Kirby & Mahoney (NeurIPS 2021) showed that PINN training on a hard PDE problem is dramatically improved by building up to it: train a network on an easier version of the problem first, then progressively warp the training target toward the hard one. This is exactly the seismic FWI frequency-continuation idea (Bunks et al. 1995) reborn for PINNs.

Why curriculum works for PINNs

The optimisation landscape of a multi-scale PINN problem has many local minima. Direct optimisation lands in the closest one to the random initialisation. Training on an easier surrogate (low-frequency target) produces a smoother loss landscape with one dominant minimum; the optimiser reliably finds it. Then warping toward the harder target moves the minimum, but only locally, the network is already in the basin of the right minimum. This is how natural humans solve hard regression problems too: start with a simple model, then add complexity.

The technique has three flavours, distinguished by what is being warped:

• Frequency continuation: warp the target frequency content from low to high. This is what Bunks 1995 introduced for seismic FWI; Krishnapriyan 2021 reused it for PINNs.
• Spatial-domain expansion: train on a small subdomain first, then enlarge. Useful for problems with sharp features at the boundary.
• Optimiser handoff: train with Adam (robust, fast initial convergence) then switch to L-BFGS (precise, second-order). Standard practice in seismic PINN papers.

Try it: frequency continuation

The widget races three strategies on the multi-scale target $f(x; w) = \tfrac{1}{2}\sin(\pi x) + \tfrac{1}{2}\sin(w \pi x)$ with the hard target at $w = 10$. Crucially the network is a vanilla 1-64-64-1 Tanh MLP, no Fourier features, no SIREN. Spectral bias (§0.9, §2.2) is what is being fought.

• naive: train directly at $w = 10$ for 2500 epochs.
• lr-anneal: same target, with cosine-annealed learning rate. (Curriculum lite, tries to escape the minimum without changing the target.)
• frequency continuation: five stages $w = 2 \to 3 \to 5 \to 7 \to 10$, each 500 epochs.

What you should observe

• naive: gets stuck on the low-frequency component, never resolves the $\sin(10 \pi x)$ feature. Final relative-L² is typically 60-80%, the spectral-bias plateau.
• lr-anneal: similar to naive at typical seeds; learning-rate scheduling alone does not introduce the right inductive bias for multi-scale targets.
• frequency continuation: the network learns the $w = 2$ component cleanly, then the $w = 3$ overlay is a small perturbation, and so on up to $w = 10$. The vertical dashed lines mark stage boundaries; the loss panel shows the curriculum descending in steps. Final relative-L² typically drops to 30-50%, a meaningful win on a target that vanilla architectures cannot fully resolve. With Fourier features in addition (§2.2), the same curriculum drives error to $10^{-4}$.

Frequency continuation for seismic FWI

Bunks et al. (1995) introduced frequency continuation for seismic FWI: filter the data to a low-pass band, invert; widen the band, re-invert; repeat. The reasoning was identical to the PINN case, low-pass data gives a convex(er) misfit with one dominant minimum, avoiding the cycle-skipping pathology of direct full-bandwidth FWI. Modern PINN-FWI methods (Song et al. 2023; Liu et al. 2024) use the same idea: start the PINN training with a low-pass-filtered version of the wavefield, then progressively widen the bandwidth.

Part 6 returns to this in detail: the curriculum is the centrepiece of how PINN-FWI avoids cycle-skipping on Marmousi-class problems.

Optimiser handoff: Adam → L-BFGS

The widget does not include L-BFGS (which is hard to implement well in pure JS), but the technique deserves mention. After Adam reaches a plateau, switching to L-BFGS for ~100 iterations typically reduces the loss by another 1-2 orders of magnitude. L-BFGS uses approximate second-order information and converges superlinearly near the minimum; Adam is gradient-based and converges linearly. The combination is the de-facto standard in published seismic PINN papers (Rasht-Behesht 2022, Song 2023, Wu 2024).

If you need L-BFGS in JavaScript, the Cephes-derived numericjs.minimize works for problems up to ~10k parameters; beyond that you need WebAssembly or native code.

References

• Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., Mahoney, M.W. (2021). Characterizing possible failure modes in physics-informed neural networks. NeurIPS.
• Bunks, C., Saleck, F.M., Zaleski, S., Chavent, G. (1995). Multiscale seismic waveform inversion. Geophysics 60(5).
• Song, C., Alkhalifah, T., Waheed, U.B. (2023). A versatile framework to solve the Helmholtz equation using physics-informed neural networks. Geophys. J. Int.