Approximability of Deep Computations

In this paper, we introduce a general notion of computation which, we contend, captures the essence of digital computations involving floating-point arithmetic. As computing power expands, so does the need for foundational frameworks to understand systems and applications, particularly their asymptotic properties as scale and scope grow potentially indefinitely; such frameworks are needed across multiple areas of computational science and engineering. A prominent example is that of neural networks, which pass computations through an increasing number of compositional layers. Deep learning systems are typically based on networks with a large number of layers (i.e., increasingly deep networks), which are correspondingly expensive to compute; it is of enormous importance to approximate the output of such deep networks more efficiently—even, simply to understand whether such approximation is possible in principle.

Notable recent frameworks working to leverage asymptotic properties of deep networks include:

• •

  Neural Ordinary Differential Equations (Neural ODEs) [CRBD18], [HLA⁺21], which introduce ODEs as mechanisms to capture residual deep networks asymptotically (as the number of layers goes to infinity), and use numerical or analytical ([HLA⁺22]) ODE solutions as “shortcuts” to approximate the output of such deep networks minus the need for multifold iterations; and

• •

  Deep Equilibrium Networks (DEQs) [BKK19], [BKK20], [APL⁺22], which model deep networks arising by iterating the same (“parameter-tied”) layer transition indefinitely (in cases when such network reaches an asymptotic equilibrium), then uses fixed point numerical solvers to shortcut the deep computation implied.

Related problems have also arisen in numerical optimization, where discrete iterative optimization algorithms—whose asymptotic properties as the step size goes to zero are of perennial interest and importance—are now modeled asymptotically using dynamical systems [SDJS22], and in control theory, where properties of parameter-dependent asymptotic computations are of central importance in key applications [CRBD18, HL⁺19, LJ23].

Such approaches to asymptotic (‘deep’) computations are fundamentally asking whether a complex composition of function applications may be realized through a smaller computation. From the perspective of this paper: “Can the result of some large (conceptually infinite) sequence of function compositions be approximated, effectively and finitarily, from accepted computational primitives (‘atomic predicates’) and standard floating-point operations?” Without precise context, the questions of whether the asymptotic limit of a computation exists, and when it can be feasibly approximated are ill-defined; in this paper, we propose a framework and basic tools that we hope will be useful in addressing such questions, among others, concerning the notion and nature of deep computations.

The remainder of this introductory section provides an informal overview of our approach and main results.

Our computations $\gamma$ are transformations accepting as input a state $v$ and returning an output state $w=\gamma(v)$. We posit that states $v$ be uniquely characterized in terms of a collection of real-valued quantities $P(v)$ (as $P$ varies over a fixed collection $\mathcal{P}$ of “primitive predicates”). We call each real value $P(v)$ a feature of $v$; it is appropriate to think of such feature as a the “$P$-th coordinate” of $v$ —indeed, a particular instance of our framework is when states are vectors $v\in\mathbb{R}^{n}$ and there is one predicate $P_{i}(v)=v_{i}$ for each of the coordinates $v_{i}$ ($1\leq i\leq n$) of $v$. Each predicate $P$ is “atomic” in the sense that it captures a primitive feature of states $v$, i.e., features of any given state are regarded as computable ab initio. De facto, a computation $\gamma$ effectively maps the collection $(P(v))_{P\in\mathcal{P}}$ of predicate values of an arbitrary input state $v$, to another such collection $(P(\gamma(v)))_{P\in\mathcal{P}}$ uniquely characterizing the output state $\gamma(v)$.

Understood in such generality, transformations $v\mapsto\gamma(v)$ can hardly be called computations in any reasonable sense. Indeed, the identification of a state with the collection of real values of its primitive predicates in any sensible paradigm for floating-point computations transforming states implies that each real-valued feature $Q(w)$ of the output state $w=\gamma(v)$ ought to depend continuously on features $P(v)$ of the input state; such continuity assumption is implied by the tenet that floating-point computations are intrinsically approximate, never quite exact. In general, one cannot expect finite-precision calculations to be able to approximate real-valued features (of the output) that vary discontinuously with respect to the real-valued quantities that one uses to encode the input state!

For simplicity, we assume the inputs $v$ and outputs $w=\gamma(v)$ of computations $\gamma$ belong to the same state space $L$ which, when endowed with the collection $\mathcal{P}$ of real-valued predicates $P:L\to\mathbb{R}$, becomes a Computation States Structure (CSS) $\underline{L}=\langle L,\mathcal{P}\rangle$.

Our results are deeper (and, hopefully, most illuminating) when the CSS is endowed with an infinite predicate collection $\mathcal{P}$. Because of the importance of such special case, and for ease of exposition, throughout the remainder of this introductory section, we assume:

$\mathcal{P}$ is countable.

The case of $\mathcal{P}$ countable is quite relevant in applications for reasons we now explain. Let the predicate collection be $\mathcal{P}=(P_{n})_{n\in\mathbb{N}}$. Every state $v\in L$ is characterized (at least from the purely structural perspective we adopt) by the real sequence $(P_{n}(v))_{n\in\mathbb{N}}$ of its predicate values, called the type $\operatorname{tp}(v)$ of $v$; thus, $L$ is effectively identified with the set of such types, i.e., with a subset $L\subseteq\mathbb{R}^{\mathbb{N}}$ of the set $\mathbb{R}^{\mathbb{N}}$ of all real sequences $(r_{n})_{n\in\mathbb{N}}$; one may think of $r_{n}=P_{n}(v)$ as the “$n$-th entry” of a state $v$. We shall require that computations $\gamma:L\to L$ have “features” (i.e., entries) $P_{n}{\circ}\gamma:v\mapsto P_{n}(\gamma(v))$ varying continuously with (the type of) $v$ for each $n\in\mathbb{N}$; explicitly, every such computation feature $P_{n}{\circ}\gamma(v)$ is required to vary continuously with respect to any feature $P_{m}(v)$ of $v$.

State types effectively encode, e.g., intermediate stages of neural networks. (For this reason, we also refer to $L$ as the “layer state space”; here, the identification of a state $v\in L$ with its type $\operatorname{tp}(v)$ is implicit.)¹¹1Neural networks whose layer transitions depend not only on the current state but possibly on earlier ones may be formalized as multi-argument transformations. An appropriate setting is that of $n$-ary CCSs, which are informally introduced in Remark 3.4(2). A different view of layer transitions that allows them to depend on additional parameters is the notion of Parametrized Family of Computations discussed in Appendix A.1.4. For simplicity, computations $\gamma$ generally represent (single-argument) maps $L\to L$ throughout this paper. A neural network of depth $m$ may be regarded as a computation $\gamma=\gamma_{m}\ldots\gamma_{2}\gamma_{1}$ composing $m$-many single-layer transitions $\gamma_{i}$ that are computations of a very specific kind. Roughly speaking, once one fixes a (suitable) nonlinear activation function $\tau:\mathbb{R}\to\mathbb{R}$ once and for all, each feature of each transition $\gamma_{i}$ is obtained by applying $\tau$ to some linear combination of input features (the chosen coefficients in forming such linear combinations are the parameters of the transition $\gamma_{i}$).

At any given stage (layer), only finitely many real-valued features of the computation are meaningful in an effective sense; however, the study of arbitrarily deep networks, and of deep layers/deep equilibria of such networks requires keeping track of numerous features the number of which is not necessarily bounded beforehand. The natural setting to treat a finite number of features possibly growing without bound is with a countable predicate collection $(P_{n})$.

We formalize the deep layers and equilibria mentioned above as deep computations (or ultracomputations), which capture a precise notion of asymptotic limit of computations. Such deep computations are obtained as pointwise limits of sequences $(\gamma_{n})_{n\in\mathbb{N}}$ of given computations, where “pointwise” means for each individual feature. (More generally, deep computations may arise as arbitrary pointwise ultralimits of computations.) Deep computations are not necessarily realizable as layer transition maps $L\to L$, but typically only as “transforms” $f:L\to\mathfrak{L}$ into the space $\mathfrak{L}$ of state types (namely $\mathfrak{L}=\overline{L}\subseteq\mathbb{R}^{\mathcal{P}}$ is the topological closure of $L\subseteq\mathbb{R}^{\mathcal{P}}$).

A tenet of our approach is that a transform $f:L\to\mathfrak{L}$ is to be considered “effectively computable” if, for each $P_{n}\in\mathcal{P}$, the output feature $\xi_{n}\coloneqq P_{n}{\circ}f:L\to\mathbb{R}$ is a definable predicate in the following sense:

Given any fixed $\varepsilon>0$ (otherwise arbitrary), the output feature $\xi_{n}(v)$ is $\varepsilon$-approximated by a continuous function $\varphi$ of finitely many input features $P_{m}(v)$, on any region $K\subseteq L$ wherein every input feature $P_{i}(v)$ remains bounded in magnitude.

(The approximating function $\varphi$ above is allowed to depend on $\varepsilon$ and $K$.)

In other words, as long as input features remain bounded and one is willing to accept an error of magnitude not exceeding $\varepsilon$, output features are continuous functions of finitely many input features.

Under a suitable Extendibility Hypothesis (as formulated in §4.2) on computations, we introduce the notion of ultracomputation not merely as a transform $L\to\mathfrak{L}$, but rather as a map $\mathfrak{L}\to\mathfrak{L}$, called a “transition-in-type (t-t)”; such a t-t $\mathfrak{f}$ is considered effectively computable if each feature $\xi_{n}\coloneqq P_{n}\circ\mathfrak{f}$ is definable. The setting of transitions-in-type implies a shift in perspective that is essential to the study of deep layers in our setting.²²2In the special case when $L=\mathfrak{L}$ is closed in $\mathbb{R}^{\mathcal{P}}$, transitions-in-type are the same as transforms, namely maps $L\to L$. However (particularly when the predicate collection $\mathcal{P}$ is infinite) such situation is rather special—it amounts to a “saturation” property of the CSS $\underline{L}$—and we do not assume it a priori.

In a computational paradigm based on floating-point arithmetic, definable transforms (or t-ts) are as effectively computable as one could hope for: after all, algorithmic implementations of such arithmetic impose a priori bounds on inputs; the specific algorithm depends on such bounds in a manner paralleling the dependence of $\varphi$ approximating $\xi_{P}$ on the bounds $r_{Q}$ and the admissible error magnitude $\varepsilon$. In this paper, we are not concerned with explicit algorithmic implementations of definable ultracomputations; our results pertain to effective computability understood as the ability to carry out such computations in principle, i.e., the existence of an algorithm (which we otherwise do not provide).

(We stress that such a notion of effective computability is relative: The distinguished predicates $P(\cdot)$ are considered computable a priori —i.e., are computational primitives.)

We show (Theorem 6.2.3) that:

1. (1)

   The approximations $\varphi$ to a definable $\xi:L\to\mathbb{R}$ may be taken to be polynomials of the input features (i.e., a definable $\xi$ is polynomially definable);

2. (2)

   $\xi:L\to\mathbb{R}$ is definable iff it extends to a function $\tilde{\xi}:\mathfrak{L}\to\mathbb{R}$; and

3. (3)

   Definable transforms $f:L\to\mathfrak{L}$ are precisely those that extend to continuous t-ts $\mathfrak{f}:\mathfrak{L}\to\mathfrak{L}$ (this is the property of extendibility mentioned above).

Extendable $f$ are continuous on $L$, but extendibility is a strictly stronger property whenever $\mathfrak{L}\supsetneq L$.³³3General criteria for a transform $f:L\to\mathfrak{L}$ to takes values in $L\subseteq\mathfrak{L}$—or, similarly, for a t-t $\mathfrak{f}:\mathfrak{L}\to\mathfrak{L}$ to restrict to a transition $L\to L$—are delicate; their study exceeds the scope of the present manuscript.

Under suitable assumptions, we prove the existence of deep iterates and equilibria (understood as transitions-in-type); see Proposition 4.7 and Theorem 5.3: deep layers and deep equilibria-in-type of neural networks exist under such assumptions.

Our results in §6 characterize definability of ultracomputations. A particular case of Theorem 6.4 is as follows.

Assume that $\mathcal{P}$ is countable and each predicate $P\in\mathcal{P}$ is bounded on $L$. Fix a set $\Delta$ of extendable computations $\gamma:L\to L$. Every ultracomputation of $\Delta$ is definable if and only if, for every predicate $P\in\mathcal{P}$ and any sequences $(\gamma_{m})_{m\in\mathbb{N}}\subseteq\Delta$ and $(v_{n})_{n\in\mathbb{N}}\subseteq L$, the following Limit Exchange identity:

holds whenever the iterated limits on the left- and right-hand side both exist. Moreover, in such case, each ultracomputation of $\Delta$ is the (pointwise) limit of some sequence $(\gamma_{n})\subseteq\Delta$. The limit is attained uniformly over any set $K\subseteq L$ of states that is feature-wise bounded (i.e., $K$ is included in a “shard” in the sense of §4.1.2).

Going farther, we define smooth (ultra)computations $\mathfrak{f}$ as those having output features $P(\mathfrak{f}(v))$ varying smoothly (i.e., differentiably) with the input features $Q(v)$. Although the study of smoothness properties of definable ultracomputations is beyond the scope of the current paper, such smoothness is implicit in applications such as training of Neural ODEs [CRBD18] and equilibrium analysis of DEQs [BKK19]. In Appendix A, we informally introduce smooth transitions-in-type. In sections §A.1.2 and §A.1.4, we outline the connections of our results, respectively, with effective computability of equilibria of DEQs, and with the training of Neural ODEs (in practice done using optimal control, for instance).

The paper is organized as follows. Section §2 is a self-contained abridged summary, for countable $\mathcal{P}$ only, of the more general results in Sections §§4–6, which apply to more general structures. Section §3.4 introduces the general notions (and examples) of Computation States Structure (CSS) and Compositional Computation Structure (CCS) (the latter being essentially a CSS $\underline{L}$ expanded with a semigroup $\underline{\Gamma}$ of extendable computations $L\to L$). We also introduce the topological spaces of types—both of states and of transitions. In Section §5, we prove (under suitable hypotheses) the existence of deep computations and of deep equilibria (the main results of §§4–5 are Proposition 4.7 and Theorem 5.3). In Section §6 we prove the aforementioned characterization of definable ultracomputations (the main result being Theorem 6.4).

We are grateful to Frank Tall for his constant guidance through the world of $C_{p}$-theory.

Readers with experience in model theory will realize that the ideas presented here are strongly influenced by the work of C. C. Chang and H. J. Keisler on continuous model theory and model theory of real-valued structures [CK66, Kei23] as well as the work of J.-L. Krivine in Banach space theory [Kri76, KM81]. We owe a great debt of gratitude to these giants for allowing us to stand on their shoulders.

This section expands on the outline in Section §1, summarizing the results of subsequent sections §3.4–§6 in the special setting of computations (and ultracomputations obtained therefrom) involving states $v$ characterized by (at most) countably many real-valued “observable features” $P(v)$; readers interested in the general framework (when states are possibly characterized by uncountably many features) should skip forward to §3.4. Proofs in this section are omitted if they are presented in subsequent sections.