The loss-balance crisis: data + PDE + BC weights

Pathology #1 in §3.1 was the loss-balance crisis: the harmonic IVP with default IC weight 1 stalled with $\mathcal{L}${\textrm{IC}} \approx 1LIC​≈1 while $\mathcal{L}${\textrm{PDE}} \to 0. This section makes that observation precise, names the underlying mechanism, and lets you hand-tune the IC weight to find a working configuration. The takeaway is uncomfortable but unavoidable: there is no a-priori-correct weight. The right value depends on the gradient magnitudes of each loss term, and those magnitudes shift as training progresses. §3.3 (NTK weighting) and §3.4 (gradient-norm balancing, SA-PINN) automate the choice.

The mathematical statement

For a multi-term loss

the gradient flow is

If $|\lambda_a \nabla \mathcal{L}_a| \gg |\lambda_b \nabla \mathcal{L}_b|$, the trajectory follows $-\nabla \mathcal{L}_a$ almost exclusively; the $\mathcal{L}_b$ term is essentially invisible to the optimiser. Wang, Teng & Perdikaris (2021) called this the gradient flow pathology. The empirical signature: one term plateaus at a non-zero value while the others decrease.

Why the right λ is not obvious

You might hope that $\lambda_{\textrm{PDE}} = \lambda_{\textrm{IC}} = 1$ — "treat the terms equally" — is a sensible default. It is not. Consider:

• $\mathcal{L}${\textrm{IC}} = (u\theta(0) - 1)^2 + (u'\theta(0))^2LIC​=(uθ​(0)−1)2+(uθ′​(0))2 is evaluated at one point. Its gradient with respect to $\theta$ has magnitude proportional to whatever $\partial u$\theta(0) / \partial \theta happens to be — depends on initialisation.
• $\mathcal{L}${\textrm{PDE}} = \frac{1}{N_c} \sum{i=1}^{N_c} (u''\theta(t_i) + \omega^2 u\theta(t_i))^2 is averaged over $N_c = 50$ collocation points and involves second derivatives. Its gradient has magnitude proportional to $\omega^2 \sim 39$, scaled by random initialisation.

For random init, the second-derivative-heavy PDE gradient is typically 10²–10³ times larger than the IC gradient at $\lambda_{\textrm{IC}} = 1$. The optimiser then sees only $\mathcal{L}_{\textrm{PDE}}$ and finds $u(t) \approx 0$ — which satisfies the PDE trivially but violates the IC.

Try it: the manual weight tuner

The widget solves the same harmonic IVP from §3.1's loss-balance pathology with a slider for $\log_{10}(\lambda_{\textrm{IC}})$ from -3 to 5. For each setting, click Train for a fresh 2000-epoch fit. Watch:

• The per-term loss histories. Both should decrease together for a good $\lambda_{\textrm{IC}}$.
• The relative-L² error against the analytic solution $u(t) = \cos(\omega t)$. Below 5% is a clean fit; 30% or more means the network found a wrong steady state.

What you should observe

• $\lambda_{\textrm{IC}} = 10^{-3}$: PDE residual collapses to ≈ 0, IC stays ≈ 1, network output $u(t) \approx 0$. Relative-L² ≈ 100%.
• $\lambda_{\textrm{IC}} = 1$ (the naive default): same failure mode, slightly less extreme.
• $\lambda_{\textrm{IC}} = 10^{2}$: sweet spot — both terms decrease together, the cosine is recovered, relative-L² drops to a few percent.
• $\lambda_{\textrm{IC}} = 10^{5}$: IC dominates, network outputs $u(t) \approx 1$ to fit the IC perfectly while ignoring the PDE. Relative-L² is again large.

The Goldilocks zone is narrow — about two orders of magnitude wide — and it sits where the gradient magnitudes of the two terms are matched, not where the loss values are matched. That is the whole game in a sentence.

Why hand-tuning is not a real fix

The widget gives you four points of evidence: bad, bad, good, bad. In a real PINN problem (Burgers, wave equation, FWI) the search space has many λs (PDE residual, IC, BC, data — sometimes split per spatial component), the gradient magnitudes shift across training, and you cannot afford to scan a 4D log-space grid of $\lambda$ values to find the Goldilocks volume. Three responses have emerged in the literature:

• Hard constraints (Part 2 §2.4). When the geometry permits, reparameterise the network so the IC and BC are identically satisfied: only $\mathcal{L}_{\textrm{PDE}}$ remains. This eliminates the loss-balance crisis at the IC/BC by removing those loss terms entirely.
• NTK-based automatic weighting (Wang, Yu, Perdikaris 2022; §3.3). Compute $\textrm{tr}(K_a)$ for each loss term and set $\lambda_a \propto 1 / \textrm{tr}(K_a)$. This rebalances the gradient magnitudes automatically.
• Gradient-norm balancing (Wang, Teng, Perdikaris 2021; §3.4). Cheap heuristic: at each step, set $\lambda_a$ proportional to the maximum-over-terms ratio of gradient L²-norms. The original 2021 paper called this the GradNorm-style fix.

The widget exposes the limitation honestly

Even at $\lambda_{\textrm{IC}} = 10^{2}$, the relative-L² error is typically a few percent — better than the failed cases, but not the $10^{-4}$-level fits that NTK-balanced weighting reaches on the same problem. The hand-tuned weight is good enough to see the principle but it is not a winning strategy. §3.3 picks up the story.

Practical rule

Whenever you write down a multi-term PINN loss, log per-term gradient norms $|\nabla_\theta \mathcal{L}_a|_2$ from the very first epoch. Print them every 100 steps. If any pair differs by more than 10², you have a loss-balance crisis. Fix it before tweaking anything else — learning rate, architecture, or sampling will not save you.

References

• Wang, S., Teng, Y., Perdikaris, P. (2021). Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM J. Sci. Comput. 43(5), A3055–A3081.
• van der Meer, R., Oosterlee, C., Borovykh, A. (2022). Optimally weighted loss functions for solving PDEs with neural networks. JCAM 405, 113887.
• Bischof, R., Kraus, M. (2021). Multi-objective loss balancing for physics-informed deep learning. arXiv:2110.09813.