LLMs are the new fire. JEPA has the potential of new electricity.

In recent times, Large Language Models (LLMs) have brought about a leap forward similar to what the discovery of fire once meant to humanity: suddenly there is a powerful tool that completely changes the world. At the same time, we also know the limitations and risks. This blog dives into that "talking fire", showing its possibilities, as well as its dangers. Next, we take a look at the potential successor to the current generation of LLMs: Joint Embedding Predictive Architecture (JEPA), which by some is already labelled as the new "electricity" that can further transform the world of AI.

Despite their strengths in generating text, LLMs also have clear limitations – especially when looking within the financial sector, in which accuracy, reliability and risk management are key. For example, they don't really 'understand' the world the way people do. They have no common sense or mental representation of physical reality. LLMs predict purely based on patterns in language and the probability derived from it, without knowing whether a sentence is logical or true; it lacks an understanding of the real world.

A direct consequence of this are so-called hallucinations: answers that sound convincing, but that are factually incorrect. Compare it to Jordan Belfort selling stocks – you hesitate, but it sounds so good you almost have to believe it. These hallucinations can cause enormous problems when put in a business context. Take, for example, the lawsuit in which an Air Canada chatbot falsely promised a discount – and the company was ultimately held liable (source). This example shows how important it is not to blindly trust generative AI systems, especially in customer-oriented environments.

Another shortcoming of LLMs is that they lack long-term memory or planning skills. An LLM generates each subsequent word without an explicit plan for the entire sentence or paragraph. This can cause small errors at the beginning of an answer, ultimately growing into an incorrect or inconsistent outcome. In short, LLMs are great pattern recognizers in the world of language, but they lack a deeper understanding of what that language actually describes in the real world. In sectors such as finance, this is a significant problem. Some progress has been made recently by adding short-term and long-term memory, but that's not enough.

In short: LLMs make a lot possible, but can be frustrating at the same time. They are useful as assistants – not as advisors. Therefore, human supervision remains crucial. Only through professional assessment and critical thinking can we guarantee the quality and reliability of the output. After all, how can I trust an LLM to properly explain quantum mechanics to me, if it doesn't even know how many times the letter R appears in the word strawberry?

LLMs have developed at lightning speed, but their limitations remain apparent. But what if we could tackle those LLM limitations? Yann LeCun (Meta), one of the 3 "godfathers of AI", highlights the fundamental shortcomings: "they don't understand how the world works and they can't remember anything, they can't reason like humans, nor can they plan" (Meta’s Yann LeCun Wants to Ditch Generative AI). That's why in 2023 his team developed JEPA: an AI model designed to let AI learn the way we humans do naturally: through experience and forming a mental model of the world. Instead of 'educated guesses' at the next word, JEPA focuses on representational learning.

At the risk of delving too much into the terminology, we can illustrate the difference between JEPAs and LLMs through a metaphor: imagine you are putting together a jigsaw puzzle and you need to find the next piece.

· The LLM approach:

Tries to guess which piece of the puzzle comes next by looking purely at shape and colour, piece by piece, without actually understanding the picture of the entire puzzle.

· The JEPA approach:

First forms a mental image of what the overall picture of the puzzle should be and then uses that to find the next logical piece of the puzzle. In other words, JEPA generates an abstract understanding of the puzzle rather than starting with a limited set of knowledge.

The first JEPA model focused on image recognition. A video version has now been added to this: V-JEPA, which simply learns by observing. Like a child who understands its environment by looking, V-JEPA can passively learn context and recognize patterns, without active input. This way of learning opens the door to AI systems that can plan and reason more effectively and are less susceptible to hallucinations.

However, JEPA is still in its infancy. It's not a one-size-fits-all model that generates ready-made texts like ChatGPT. Additional components are needed to create useful output. But the potential is enormous.