Mathematics > Numerical Analysis
Title: Phase Transition from Random Features to Fully Trainable Networks
Abstract: Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $\alpha>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((\kappa_P\log(m/\delta)/m)^\alpha)$ for two-layer networks of width $m$ and trainable proportion $P$, where $\kappa_P=(P/\kappa_1^*+(1-P)/\kappa_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-\delta$. The constants $\kappa_0^*$ and $\kappa_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $\alpha=(d+2)/(d-1)$ and $\kappa_0^*=4\sqrt{2}\tau(d+1)/d$ for initialization scale $\tau\ge1$, and $\kappa_1^*=1$ under ridge leverage sampling. For $d=5$ and $\tau=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
Submission history
Access Paper:
Current browse context:
References & Citations
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .