Private Inference on Neural Networks: Approximating Non-Linearities Under FHE

Abstract

This article works through what it takes to run a neural network on encrypted input. The linear layers — matrix multiplications and convolutions — are sums of products, exactly the operations homomorphic encryption performs natively and batches efficiently, so they map cleanly onto CKKS. The obstacle is the non-linearities: ReLU, GELU, sigmoid, and softmax are not polynomials, yet addition and multiplication are all FHE offers, so every activation must be either approximated by a polynomial or evaluated through a comparison-capable mechanism like TFHE's programmable bootstrapping. It examines the approximation strategies — minimax and Chebyshev polynomials over a bounded input range, replacing activations with FHE-friendly functions such as the square at training time, and hybrid schemes that switch to TFHE for exact comparisons — and the accuracy-versus-depth trade-off that governs them, since a higher-degree polynomial fits better but adds multiplicative depth and thus bootstrapping cost. It closes with an honest read on feasibility: small and medium networks run privately today at a latency premium, softmax and very deep transformers remain the frontier, and progress tracks better approximations and hybrid protocols. The theme: private inference is the art of getting a network's non-linearities through a system that can only add and multiply.

Running a neural network on encrypted data splits cleanly into an easy part and a hard part. The easy part is the linear algebra — the matrix multiplications and convolutions that make up most of a network are sums of products, precisely what homomorphic encryption does natively. The hard part is everything non-linear: the activation functions and the softmax, which are not built from addition and multiplication and so cannot be evaluated directly under a scheme that offers only those. Private inference is, to a first approximation, the engineering of getting those non-linearities through an arithmetic-only system, and how well it is done sets the boundary of what confidential inference can achieve.

The linear layers are natural

Most of the computation in a neural network is linear: a fully connected layer is a matrix-vector product, a convolution is a structured set of multiply-accumulates, and both are nothing more than sums of products. These are exactly the operations homomorphic encryption performs directly, so the linear layers of a network translate onto FHE with no approximation at all. Under CKKS, whose approximate real arithmetic matches the real-valued weights and activations of a network, a linear layer is a natural fit.

Batching makes this efficient rather than merely possible. Because a CKKS ciphertext packs many real values into its slots, the activations of an entire layer can live in one ciphertext, and a single homomorphic operation can act across the whole layer in parallel. The linear-algebra structure of a network — large tensors transformed by matrix multiplications — aligns with the batched, single-instruction-multiple-data character of FHE, so the linear portion of inference is where FHE is at its most comfortable.

The pipeline of a network alternates linear layers with non-linear activations, and it is worth seeing the shape plainly: input, then a linear layer, then a non-linearity, then another linear layer, and so on to the output. The linear steps pass through FHE cleanly; it is the non-linear steps sitting between them that break the arithmetic-only model. Those are where the difficulty, the cost, and the interesting engineering all concentrate.

Linear layers map cleanly onto FHE arithmetic; the non-linearity between them is the step that must be approximated.

The non-linearity problem

Neural networks depend on non-linear activation functions — ReLU, GELU, sigmoid, tanh — and on the softmax that turns logits into a probability distribution. These functions are what let a network model complex relationships; a network of only linear layers collapses to a single linear map. But none of them is a polynomial, and a homomorphic scheme can compute only additions and multiplications, which build only polynomials. There is a fundamental mismatch between what a network requires and what FHE natively provides.

The mismatch is sharpest for functions defined by a decision or a transcendental operation. ReLU is a comparison — the maximum of the input and zero — which arithmetic cannot express directly. Sigmoid and GELU involve exponentials; softmax involves exponentials and a division, and division is itself not a native FHE operation. Each of these must be reconstructed out of the only tools available, and the quality and cost of that reconstruction is the crux of private inference.

There are two broad ways across the gap. One is to approximate the non-linear function with a polynomial, which FHE can evaluate, accepting some error. The other is to leave the arithmetic schemes for the non-linear step and use a mechanism that can compute comparisons and lookups directly — the programmable bootstrapping of the TFHE family, which evaluates an arbitrary function through the bootstrap itself. Real systems use one or both, and the rest of this article is about the trade-offs each entails.

FHE natively provides only addition and multiplication; networks need comparisons and transcendental functions that must be reconstructed.

Approximation strategies

The polynomial route replaces each activation with a polynomial that fits it closely over the range of inputs the activation actually sees. Minimax and Chebyshev approximations give the best fit for a given degree, and restricting the input domain — which is reasonable because activations in a trained network occupy a bounded range — lets a modest-degree polynomial track the true function well. The polynomial is then evaluated with the additions and multiplications FHE provides, no comparison needed.

The governing trade-off is accuracy versus depth. A higher-degree polynomial approximates the function more faithfully, but it has greater multiplicative depth, which spends more noise budget and forces more bootstrapping — the dominant cost from the previous article. So the approximation cannot simply be made as accurate as desired; its degree is bounded by the cost one can afford, and the art is finding the lowest-degree polynomial whose error the model tolerates. A second strategy sidesteps approximation at inference time by changing the model: replacing ReLU with an FHE-friendly activation such as the square function during training, so the network learns to work with a function FHE evaluates exactly and cheaply.

The hybrid strategy uses the right tool for each part. It keeps the linear layers in a batched arithmetic scheme like CKKS and switches to a TFHE-style programmable bootstrapping for the non-linear steps, where an exact comparison or an arbitrary lookup is needed — evaluating ReLU or a table-based activation precisely rather than approximately. The cost is the machinery to convert between schemes, but the payoff is exactness where polynomial approximation would be expensive or inaccurate. The table summarizes the strategies and their trade-offs.

\[f(x) \;\approx\; \sum_{i=0}^{d} c_i\, x^{i}, \qquad \text{multiplicative depth} \;\approx\; \lceil \log_2 d \rceil\]

\[\text{higher } d \;\Rightarrow\; \text{lower error, greater depth, more bootstrapping}\]

Strategies for evaluating non-linearities under FHE.
Strategy	Idea	Trade-off
Polynomial approximation	fit activation with a bounded-degree polynomial	accuracy versus multiplicative depth
FHE-friendly training	replace ReLU with square/low-degree activation	retrain model; some accuracy loss
Hybrid CKKS + TFHE	programmable bootstrapping for the non-linear step	exactness at the cost of scheme switching

What is feasible, and the AI stakes

The honest picture is a moving frontier. Small and medium networks — classifiers, moderate convolutional and fully connected models — can be evaluated on encrypted inputs today at a real but tolerable latency premium, especially when the activations are polynomial-approximated or the model is trained to be FHE-friendly. These are genuine deployments, not just demonstrations, and they cover a meaningful set of privacy-sensitive tasks in health, finance, and enterprise document processing.

The frontier is the large transformer. Softmax over attention, GELU activations, and the sheer depth and width of a modern language model stack up multiplicative depth and non-linearities far beyond what small models require, so fully homomorphic inference on a large LLM at interactive latency is not yet practical. Research pushes on every front at once — better low-degree approximations of softmax and GELU, hybrid protocols that combine FHE with other secure-computation techniques, and hardware acceleration for the bootstrapping that dominates the cost — and the boundary of what is feasible advances with each.

For confidential AI the stakes are exactly at this boundary. The prize is inference on a hosted model that never sees the plaintext prompt, and the thing standing between today and that prize is the cost of getting a network's non-linearities through an arithmetic-only system. Every advance in approximating softmax cheaply, in FHE-friendly model design, and in hybrid evaluation moves a larger class of models across the line from impractical to deployable. Reading private inference this way — as the engineering of non-linearities under a scheme that can only add and multiply — is what makes sense of both its current limits and the direction of its progress, and it is why the linear layers were never the story: the activations always were.

⚠️

The non-linearities are the constraint. Linear layers pass through FHE cleanly, so private-inference cost and feasibility are set by how the non-linearities are handled: polynomial approximation trades accuracy against depth, FHE-friendly training changes the model, and hybrid CKKS+TFHE buys exactness at the price of scheme switching. Small nets are practical now; large transformers are the frontier.

Key takeaways

A network's linear layers are sums of products — native, batchable FHE operations that map cleanly onto CKKS with no approximation.
Activations (ReLU, GELU, sigmoid) and softmax are non-polynomial, but FHE offers only addition and multiplication, so every non-linearity must be reconstructed.
Polynomial approximation fits activations over a bounded range; accuracy rises with degree but so does multiplicative depth and bootstrapping cost.
FHE-friendly training replaces ReLU with cheap functions like the square, and hybrid CKKS+TFHE uses programmable bootstrapping for exact non-linear steps.
Small and medium networks run privately today at a latency premium; softmax and large transformers remain the frontier.
Private inference is fundamentally the engineering of non-linearities under an arithmetic-only scheme, and progress tracks better approximations, FHE-friendly models, and hybrid protocols.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

✅Planning private inferencechecklist

Questions that decide feasibility.

Which activations does the model use, and over what input range?
Can each activation be approximated by a low-degree polynomial within tolerance?
Would replacing ReLU with an FHE-friendly activation and retraining help?
Is a hybrid CKKS+TFHE design worth the scheme-switching cost for exact non-linearities?
Does the model's depth keep the bootstrapping count within the latency budget?

🚀The private-inference mental modelquickstart

Three facts to hold onto.

Linear layers are free-ish; non-linearities are the cost and the constraint.
Polynomial approximation trades accuracy for depth (and bootstrapping).
Small nets are practical now; large transformers are the frontier.

🔒Confidential-inference design stancepolicy

A stub recording the non-linearity plan.

private_inference:
  linear_layers:
    scheme: ckks
    batched: true
  non_linearities:
    strategy: [polynomial_approx, fhe_friendly_train, hybrid_tfhe]
    accuracy_vs_depth: budgeted
  feasibility:
    small_medium_nets: practical
    large_transformers: frontier

Illustrative documentation template, not a product config.

Glossary

Private inference: Evaluating a neural network on encrypted input so the model provider never sees the plaintext prompt or result.
Linear layer: A matrix multiplication or convolution — a sum of products — that FHE evaluates natively and batches efficiently.
Non-linearity / activation: A function like ReLU, GELU, sigmoid, or softmax that is not a polynomial and cannot be evaluated directly by arithmetic-only FHE.
Polynomial approximation: Replacing a non-linear function with a bounded-degree polynomial that FHE can evaluate, trading accuracy against multiplicative depth.
FHE-friendly activation: A cheap, low-degree function such as the square, substituted for ReLU at training time so inference is exact and shallow.
Programmable bootstrapping: A TFHE mechanism that evaluates an arbitrary function (e.g., a comparison or lookup) through the bootstrap, used for exact non-linear steps in hybrid designs.