R. J. Lipton tells the following story about TeX:

Jeff Ullman moved in the late 1970’s from Princeton University to Stanford University, where he is still. […] he wanted to try the then new system that Knuth had just invented […] He got some basic info on TeX from one of Knuth’s students and tried it out. He liked the output […] So Jeff went to see Knuth and asked him for the grammar for TeX. Knuth looked at him and said,

What grammar? There is no BNF, there is no description.

Jeff was shocked. Knuth was famous for his research into the area of formal grammars, he had invented LR parsing in 1965. How could he design a language without any formal grammar? Well he did. The legal inputs to TeX where those strings Knuth’s program accepted—no more no less.

There is no grammar for TeX in general, and the answers to this question show why.

But if you are one of the people interested in TeX only for the ability to typeset mathematics, and do not care about the bulk of TeX—typesetting of text (as in “breaking paragraphs into lines”), tables, macros, etc.—then you may be interested in learning only about that part of TeX.

There are other programs that reproduce this to varying extent: MathJax and KaTeX (for rendering to web), iosMath and AndroidMath (for rendering to iOS or Android native views, without using a web view–for which you could use MathJax / KaTeX).

If you’re trying to understand by yourself what TeX does, then it’s all documented in The TeXbook, which is a manual and guide for the TeX program, especially Appendix G. (The source code of the program is well-documented and available as book, and even online – but reading the source code written in the style of Pascal programs of the 1970s, that too by an unconventional programmer who has developed his own distinctive style and conventions, needs quite a bit of work.)

You may want to read other sources before The TeXbook. Here are some related questions on TeX.SE 1, 2, 3, 4, 5, and there’s a great article called Appendix G Illuminated (TUGboat 2006). A good (and free) reference is TeX by Topic.

Finally, someone’s tried to actually formalize TeX’s layout of mathematics, in the paper:

reimplementing it in SML. There’s actually a bit more detail that I forgot to include here; see this answer.