SIREN — sinusoidal representation networks

Vincent Sitzmann and collaborators introduced SIREN ("implicit neural representations with periodic activation functions") in late 2020 as an alternative architectural fix to the spectral-bias problem. Where Fourier-feature MLPs (§2.2) put the periodic structure in a fixed input embedding and use a vanilla MLP afterward, SIREN puts it everywhere: every hidden activation is $\sin$. With the right initialisation, this produces a network that natively represents oscillatory functions, with no Fourier-feature precomputation needed.

The architecture

For a network $u(\mathbf{x})$ with hidden layers indexed by $i$:

The $\omega_0$ multiplier inside the activation is the central design choice. Its purpose is to set the magnitude of the pre-activation feeding $\sin$: too small and $\sin$ behaves like its linear approximation (network is essentially vanilla); too large and the network is wildly oscillatory at initialisation. The default in the paper is $\omega_0 = 30$.

This textbook's runtime bakes the $\omega_0$ scaling into the weight initialisation rather than applying it inside the activation (the architectures end up equivalent in distribution — the difference is bookkeeping, not behaviour). The first layer is initialised with $W \sim \mathcal{U}(-\omega_0/n,,\omega_0/n)$; hidden layers with $W \sim \mathcal{U}(-\sqrt{6/n},,\sqrt{6/n})$. This combination is what Sitzmann et al carefully derived to keep activations roughly $\mathcal{N}(0, 1)$ through the entire stack.

Try it

Four SIREN networks race side by side, identical except for $\omega_0 \in {1, 5, 30, 60}$. The default target is a five-frequency cosine sum (frequencies 1, 3, 7, 12, 20 cycles per [-1, 1]) — too high-frequency for a vanilla MLP to handle in reasonable time. Press Play:

• ω₀ = 1: SIREN behaves like a vanilla sin-activation MLP. Spectral bias still bites; only the lowest 1–2 components are recovered.
• ω₀ = 5: middle frequencies emerge. The 12 and 20 cycle components are still missing.
• ω₀ = 30: matched to the highest target frequency. Clean fit across the whole spectrum within ~1500 epochs.
• ω₀ = 60: well above the target frequencies but still trains cleanly at the conservative default lr (1e-3) — empirically reaches comparable error to ω₀ = 30. The instability the literature warns about appears at higher learning rates: try sliding lr to 1e-2 and watch ω₀ = 60 spike or diverge while ω₀ = 30 stays stable. Sitzmann's recommendation of ω₀ = 30 as a default is about robustness across learning rates, not raw fit quality.

SIREN vs Fourier features

Both architectures attack the same problem (spectral bias) and have a similar single-knob hyperparameter (σ for Fourier, ω₀ for SIREN). The qualitative differences:

• SIREN is more expressive per parameter for high-frequency targets, because every layer is oscillatory. A Fourier-feature MLP only has the embedding layer doing oscillation; the MLP itself is still a vanilla Tanh stack with its own spectral bias inside the embedded space.
• SIREN is more sensitive to initialisation. The Sitzmann init is precisely tuned; deviations break training. Fourier-feature MLPs use ordinary (Xavier or He) init for their MLP block and are more forgiving.
• Fourier features can be combined with any backbone — Tanh, Swish, even ReLU on the MLP side. SIREN is sin-only by design.
• SIREN gives smoother input-derivatives, which matters for PINN problems where the residual involves $\partial^2 u/\partial x^2$. Fourier features feed into a Tanh MLP whose own derivatives can be slightly less clean.

In production seismic-PINN code, both choices appear. Smith, Azizzadenesheli and Ross (EikoNet 2021) use Fourier features. Bin Waheed et al (2021) and Song et al (2021) use SIREN-like networks for high-frequency Helmholtz problems. The decision is usually made by an empirical sweep on a small test problem before committing to the production architecture.

Why this matters for seismic PINNs

Seismic wavefields at exploration frequencies (10–80 Hz) on multi-km targets are extreme high-frequency-representation problems. A vanilla Tanh MLP simply cannot represent them at any reasonable depth or width; spectral bias makes training stagnate at a low-frequency blur. SIREN with $\omega_0 = 30$ or Fourier features with $\sigma \sim$(target frequency × domain length) is the standard fix in modern PINN-FWI papers. We will see this reused throughout Parts 4 (Helmholtz), 6 (velocity inversion), and 7 (eikonal).

Pause-and-check. (1) Slide the lr from 1e-3 (default) up to 1e-2 and watch what happens to the ω₀ = 60 panel. What is the failure mode? (2) Why does the SIREN initialisation use different bounds for the first layer vs hidden layers? (3) For an inverse problem where you do not know the wavefield's frequency content in advance, would you choose SIREN with a fixed ω₀, Fourier features with a fixed σ, or one of the multi-scale architectures of §2.5? Defend the choice.

References

• Sitzmann, V., Martel, J.N.P., Bergman, A.W., Lindell, D.B., Wetzstein, G. (2020). Implicit neural representations with periodic activation functions (SIREN). NeurIPS.
• Tancik, M., Srinivasan, P.P., Mildenhall, B., et al. (2020). Fourier features let networks learn high-frequency functions in low-dimensional domains. NeurIPS.
• Wang, S., Wang, H., Perdikaris, P. (2021). On the eigenvector bias of Fourier feature networks. CMAME 384, 113938.
• Huang, X., Alkhalifah, T. (2022). PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting. JGR Solid Earth 127.