IDK, the more I learn the more it seems to me that Fourier transform is reality's cheat code. It keeps showing up everywhere.
Like, the other day I learned[0] that if you shine a light through a small opening, the diffraction pattern you get on the other side is basically the Fourier transform of the aperture outline.
(Yes, this also implies that if you take a Fourier transform of an image and make a diffraction grating off the resulting pattern, projecting light through it should paint you the original image.)
Right, physics runs on differential equations and sinusoids/exponentials are eigenfunctions of differential equations.
You can project reality onto any complete basis of functions you like, but this one tends to diagonalize the physics of our universe, which is an overpowered ability inside of our universe.
> tends to diagonalize the physics of our universe
Because it diagonalizes all good translationally invariant operators, and our universe is fond of translation invariance until you get into general relativity. (This sounds less mysterious once you learn that all good translationally invariant operators are essentially convolutions. Neither of these statements is often taught at the elementary level, probably because of the difficulty and ambiguity in defining “all good” and “are essentially”.)
> good translationally invariant operators are essentially convolutions.
I first learned this seemingly obvious in hindsight corollary from another comment on HN [1] and it blew my mind. I wish it was included in the usual descriptions of why we choose complex exponential basis for things like the Laplace transform. It's all well and good that they're eigenfunctions of translations, but it still left me wondering why we care about that in the first place.
(If your first exposure is instead from a physics or EE perspective, I suspect the framing would be more obvious, as compared to how it's usually introduced in DiffEq when the choice of basis just seems like a "neat" trick given that it behaves well under differentiation).
> It’s all well and good that they’re eigenfunctions of translations, but it still left me wondering why we care about that in the first place.
You don’t need the convolution statement to see that (as the comment you linked above also demonstrates). A good linear algebra course should have the statement that any set of commuting operators has a common eigenbasis[1]. In particular, if an operator has nondegenerate eigenvalues, then its (essentially unique) eigenbasis is also an eigenbasis for any operator that commutes with it. Take a translation as the former operator and any translation-invariant operator as the latter and you see why all of these just got diagonalized simultaneously.
Above I’ve blatantly ignored all the infinite-dimensional problems that arise when attempting to explain grown-up Fourier transforms, but literally this is actually enough if your Fourier transform is finite-dimensional—the Fourier transform on Z/nZ (aka the discrete Fourier transform on a circle) is most commonly used in applications, but literally everything goes through word for word on an arbitrary finite Abelian group. If your mental powers of abstraction feel like they should be able to acquire some intuition about the real case from the finite case, I highly recommend you read Paul Garrett’s note on the topic[2].
That said, yes, the statement on convolution operators is unreasonably hard to find or stumble upon in the literature. Part of it is that stating it properly is annoying and fussy. Another part is purely terminological: the term “convolution operator” is really rare among books younger than half a century. The usual term is instead “Fourier multiplier” or just “multiplier”, which basically makes sense iff you already know the convolution theorem. Searching for the modern term should give you a plethora of sources. (AFAIU, part of the motivation for this switch is that working with the Fourier transform of your convolution kernel instead of the kernel itself allows one to avoid distributions / generalized functions—and the associated hardcore functional analysis—longer. Consider that, if you want to use kernels, already the literal identity operator forces you to work with the Dirac delta.)
[1] If A and B commute, then any eigenspace ker(A-tE) of A is invariant under B, so decompose your space into a direct sum of eigenspaces of one of your operators and recurse on each. Choose an arbitrary basis when the operators run out.
(Almost-)linear models do linear things, it seems, and the Fourier transform is the quintessential linear thing.
It is also an extremely neat piece of the real world, but I’m hesitant to guess your background and offer an explanation because your phrasing makes me suspect an engineering one. With concepts usually being the first to be culled in a course targeted at engineers, there could be quite a bit of concept debt to pay off before I could really offer something I could honestly call an explanation.
Have you tried the 3Blue1Brown video on the topic[1]? It does not AFAIR offer any answers as to why the Fourier transform should exist or be useful, but it does show very well what it does in the immediate sense.
Thanks for your reply. I think I'm going to have to start reading up on physics/differential equations. My linalg is ok but quite a bit of my computing background has been "here, this is how you calculate it" instead of concepts. I really feel theres something about Fourier that seems pretty important.
You gotta be in the in-crowd to understand that this paper, like so many others, is one of those dumb posthoc analogy/metaphor papers. These papers are where they just ran a bunch of experiments (ie just ran the training script over and over) and formulated a hypothesis empirically. Of course in order to lend the hypothesis some credibility they have to make an allusion to something formal/mathematical:
> Fourier features -- dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain
