Echo(JEPA) and Its Origins

How and by who?

Yann LeCun, JEPA's originator

JEPA (the Joint-Embedding Predictive Architecture) is the brainchild of Yann LeCun, formerly Meta's Chief AI Scientist. His vision is to create machines that can learn internal models of how the world works.

He argues the current generative models and training them with tasks to try to reconstruct every detail of a signal is wasteful and impossible since the actual world is inherently unpredictable and noisy(like LLMs with their pretraining step with next predicition token task).

He draws inspiration of how humans understand the world around them and learn from the environment in terms of abstract and high level representations and ignoring low level details most of the time.

He encourages to build strong foundational models around this thesis and these foundational models will have abstract knowledge of the environment, physics and planning . Later more specialized models can be built on top of this like action model for robots or text generation models like current LLMs.

What JEPA actually predicts

In JEPA, a model ingests a pair of related inputs;e.g. consecutive video frames or adjacent image patches and encodes each into an abstract representation. A predictor module then tries to predict the representation of the "target" input from the representation of the "context" input, bringing them closer in embedding space.

Unlike generative models, JEPA does not attempt to reconstruct every detail of the input; it works in an abstract embedding space, which lets it focus on high-level, essential information and ignore irrelevant or unpredictable details. The model can be viewed as an Energy-Based Model (EBM) operating on representations: it assigns low energy when the predicted representation matches the actual target representation, and high energy when they mismatch. The "joint embedding" part means both inputs are mapped into a common representation space where the prediction is made, rather than directly predicting raw data.

From an information-theoretic perspective, the goal is to capture as much predictable information as possible in the representations while discarding unpredictable noise. This involves a delicate balance between information preservation and compression: if the representation preserves nearly all information from the input, it may include lots of irrelevant or random detail that makes prediction difficult; if it compresses too aggressively, it may lose the structure needed to predict the target. In other words, JEPA seeks an abstraction level where the representation has high mutual information with both the input and the target, but low entropy in terms of irrelevant bits. LeCun's own example is video prediction: trying to predict every pixel of future frames is nearly impossible due to chaotic details like flickering leaves or textured surfaces.

Self-supervised learning methods

JEPA didn't invent self-supervised learning; it's a deliberate third option next to two established families, each having its drawback.

Invariance-based (joint-embedding) methods

Train an encoder to output similar embeddings for different views of the same input, views constructed via hand-crafted data augmentations (random scaling, cropping, color jittering). The energy landscape is flat for compatible inputs (low energy regardless of what the encoder outputs), so these methods need tricks to prevent representation collapse, where the encoder ignores the input entirely.

Limitation: the hard-coded augmentation invariances may not transfer across tasks or modalities; image classification and segmentation don't need the same invariances.

Generative (reconstruction) methods

Learn to directly reconstruct signal y from a compatible signal x, using a decoder conditioned on a latent z. Reconstruction collapse isn't a concern here, since the informational capacity of z is kept low.

Limitation: because the loss is in pixel/token space, the model is penalized for every low-level mismatch, incentivizing it to model texture and noise rather than semantics.

I-JEPA: the first concrete implementation

I-JEPA (Image Joint Embedding Predictive Architecture) is the first concrete implementation of JEPA, for computer vision. The idea is to predict missing information in an abstract representation that's more akin to the general understanding people have, rather than in raw pixel space.

Compared to generative methods that predict in pixel/token space, I-JEPA uses abstract prediction targets, for which unnecessary pixel-level details are potentially eliminated, leading the model to learn more semantic features. A second core design choice guiding I-JEPA toward semantic representations is its multi-block masking strategy: predicting large blocks containing semantic information (at sufficiently large scale), using an informative, spatially distributed context.

Put simply: I-JEPA patchifies the input image, then masks out everything except one large, contiguous context block, and encodes that block with the context encoder. In parallel, the target encoder, an EMA copy of the context encoder that acts as its stable, slowly-updated twin, encodes the full image and produces representations for four independently sampled target blocks.

The predictor is then asked to predict each target block's representation using only the context encoder's representations, with the target encoder's actual representations serving as ground truth.

The way I read it: the two encoders' job is to compress the image down to high-level, semantically relevant representations, and asking the predictor to fill in information about most of the image from a limited context is what injects a generative, fill-in-the-blank pressure into the training objective.

Architecture 1: I-JEPA Architecture

The figure above shows the concept; the actual forward pass moves through five components, each with its own tensor shapes. This is the original ViT-H configuration from the I-JEPA paper.

Target encoder EMA and the loss

• At the start of training, the target encoder is a direct weight copy of the context encoder: every parameter tensor duplicated (patch projection, positional embeddings, all 32 transformer layers).
• Each training step has two phases: the context encoder gets a gradient update, then the target encoder gets an EMA update:
  $$\theta_{\text{ema}} \leftarrow m \cdot \theta_{\text{ema}} + (1 - m) \cdot \theta_{\text{context}}$$
• θ_ema is the current theta value, m is the momentum, θ_context is the new context-encoder value. With m = 0.996, that's a weighted average of all past context-encoder states, with exponentially decaying weights for older states:
  $$\theta_{\text{ema}} \leftarrow 0.996 \cdot \theta_{\text{ema}} + 0.004 \cdot \theta_{\text{context}}$$
• Momentum doesn't stay fixed: it ramps linearly from 0.996 to 1.0 over the full training run.
  • Early in training (m = 0.996, a window of roughly 250 steps): the target encoder can change quickly to escape its random initialization, so targets aren't permanently anchored to noise.
  • Late in training (m → 1.0, window → infinity): representations have become semantic and stable, and freezing the targets gives the predictor a clean, consistent signal to converge on.
• The loss is the average L2 distance between each predicted patch-level representation and the corresponding target patch-level representation:
  $$\frac{1}{M}\sum_{i=1}^{M} D(\hat{s}_y(i), s_y(i)) \;=\; \frac{1}{M}\sum_{i=1}^{M}\sum_{j \in B_i} \left\| \hat{s}_{y_j} - s_{y_j} \right\|_2^2$$
• The parameters of the predictor (φ) and the context encoder (θ) are learned through gradient-based optimization, while the parameters of the target encoder (θ̄) are updated only via the exponential moving average above.

Journey to EchoJEPA

How did JEPA reach Echo? It's a long journey, from 2022 to 2026: from images, to motion, to video, to adapting video pretraining for echocardiography.

1. 2022

   A Path Towards Autonomous Machine Intelligence

   LeCun's vision for next-generation AI, with JEPA as a central component. Proposes a six-module architecture (perception, world model, cost, memory, action, configurator) and argues that a non-generative, joint-embedding world model can learn hierarchical representations of the world.

2. 2023

   I-JEPA: images

   The first concrete implementation of JEPA for computer vision.

3. 2023

   MC-JEPA: motion + content

   Extends JEPA to video by jointly learning two kinds of representations: one for static content (objects/appearance) and one for motion (optical flow).

4. 2024

   V-JEPA: revisiting feature prediction for video

   The first large-scale video-based JEPA model. Trains on a massive set of 2+ million unlabelled videos, using only the feature prediction objective: no contrastive pairs, no text or labels, no pretrained image encoder.

5. 2025

   Emergent results: intuitive physics from video prediction

   A V-JEPA model can develop a rudimentary "intuitive physics" understanding; it emerges from self-supervised pretraining on natural videos.

6. 2025

   V-JEPA2

   The scaling step: a ViT-g/16 (~1B params) trained on VideoMix22M (~22 million videos), with 3D RoPE positional encoding and a fixed EMA momentum. Its recipe (architecture, masking, and loss) is inherited directly by EchoJEPA.

7. 2026

   EchoJEPA: the heart

   A foundation model trained on 18 million echocardiograms across 300K patients, representing the largest pretraining corpus for this modality to date. By leveraging a latent predictive objective, EchoJEPA learns robust anatomical representations that ignore speckle noise.

What changed from I-JEPA to V-JEPA 2 to EchoJEPA

From patches to tubelets

V-JEPA 2 uses tubelets: small 3D cuboids that span 2 frames temporally and 16×16 pixels spatially. A single Conv3d(3, embed_dim, kernel=(2,16,16), stride=(2,16,16)) implements this: identical idea to I-JEPA's Conv2d patch projection, just with one extra dimension.

A second, standard-pretraining worked example from the deck, at 256×256 resolution:

Each tubelet passes through the Conv3d patch embedder and is projected to the encoder width (e.g., 1408 for ViT-g). The resulting token sequence is (2048, 1408).

EchoJEPA inherits this tubelet embedding unchanged from V-JEPA 2. Echocardiogram clips are tokenized with the exact same 3D convolution, just at echo's own resolution and frame rate.

3D RoPE

I-JEPA uses a fixed 2D sin-cos positional encoding, added once at the input and never touched by gradients. V-JEPA 2 replaces this with 3D Rotary Position Embeddings (RoPE), applied inside every attention layer rather than added once upfront. The extra axis encodes position along time (T), in addition to height (H) and width (W), so the model can tell "the same patch, a different frame" apart from "a different patch, the same frame".

EchoJEPA inherits 3D RoPE unchanged from V-JEPA 2. The same T/H/W structure applies directly to echo video clips, so no changes are needed here.

Masking and scaling in EchoJEPA

The masking mechanism itself, multiblock tube masking, holding the same spatial rectangle constant across every frame so the model must infer what happens inside it over time, is inherited unchanged from V-JEPA 2. What EchoJEPA does re-tune are the scale parameters, adapted to the physical constraints of an ultrasound fan:

• Context mask scale: 0.5–1.0 of the frame, wider than the default, to preserve enough visible anatomy inside a fan-shaped echo frame where much of the rectangular image is black background outside the sector
• Aspect ratio augmentation: 0.9–1.1, far narrower than I-JEPA's 0.75–1.5, respecting the fan's fixed geometry rather than sampling arbitrarily elongated crops
• Input resolution: 112×112 to 224×224, lower than V-JEPA 2's 256/384, since echo images are inherently lower-resolution than natural video
• Temporal resolution: 4–24 fps, variable per clip, since heart rate varies across patients, unlike V-JEPA 2's fixed fps

I-JEPA vs. V-JEPA 2 vs. EchoJEPA, axis by axis

The core algorithm doesn't change across these three. Every difference below is either a domain upgrade (image → video → cardiac video) or a scaling decision.

Table 6. The one conceptual addition V-JEPA 2 makes over I-JEPA: its tube masks hold the same spatial rectangle constant across every time step, forcing the model to infer what happens inside a region over time from the surrounding context, learning motion and dynamics, not just static appearance.

Why echocardiography breaks every existing foundation model

Ultrasound speckle is structured noise, not signal. It shows up as a grainy, seemingly random texture caused by constructive and destructive interference between the acoustic wavefront and tissue scatterers too small to resolve individually. Two frames of the exact same heart, captured a millisecond apart, show identical anatomy and completely different speckle. It's noise that regenerates itself every frame.

That single fact is enough to break most of the standard foundation-model recipes when they're pointed at echo:

EchoJEPA's answer

EchoJEPA applies the Joint-Embedding Predictive Architecture to echocardiography video. Instead of reconstructing pixels, it predicts the representation of masked spatiotemporal regions from visible context, entirely in latent space.

Why this solves the speckle problem: the EMA target encoder is updated slowly (an exponential moving average of the context encoder). Its representations are stable averages over many gradient steps; speckle, which is i.i.d. noise per frame, averages out. Anatomically stable structures (chamber walls, valve motion, geometry) are reinforced because they're consistent across time and views.

Contributions of the paper

• EchoJEPA. A foundation model using latent prediction pretrained on 18 million videos across 300K patients, the largest echocardiography corpus to date, achieving state-of-the-art performance on LVEF estimation and right ventricular systolic pressure (RVSP) prediction, demonstrating that latent prediction outperforms pixel reconstruction for ultrasound.
• Unified evaluation protocol. A standardized benchmark with frozen backbones, identical probes, and consistent hyperparameter search across all baseline models, enabling fair comparison of representation quality.
• Robustness benchmarks. Physics-informed perturbations using depth attenuation and acoustic shadow, revealing that EchoJEPA degrades 86% less than EchoPrime under acoustic perturbations.
• Public release. EchoJEPA-L, the first open-source JEPA-based echocardiography foundation model, trained on MIMIC-IV-Echo, is open-sourced alongside the evaluation framework at github.com/bowang-lab/EchoJEPA.
• Scale. EchoJEPA-G is pretrained on 18.1 million echocardiogram videos across 300,000 patients (the largest echo pretraining corpus assembled to date) behind a ViT-G encoder at 1.1B parameters. A smaller, publicly released variant, EchoJEPA-L, is pretrained on the 525K videos in MIMIC-IV-Echo.
• Multi-view probing framework. A method using factorized video stream embeddings and attention masking to integrate information across echocardiographic views without view-specific components. A single echocardiography study isn't one video; it's a collection of clips acquired from different transducer positions (Apical 4-chamber, Parasternal long/short axis, Subcostal, and more). The encoder processes each clip on its own, with no awareness that the others exist. A separate study-level attentive probe is what fuses them: it takes the per-clip embeddings from every acquired view and combines them with cross-attention into one study embedding.

  Right ventricular systolic pressure is the clean test case for whether that fusion is doing real work, because it is structurally a two-view measurement:

  RVSP = 4 × (TR jet velocity)² + RA pressure
             └─ Apical view ──┘   └── Subcostal IVC ──┘

Why a standardized probe is required

This is the design choice that makes the benchmark fair: every model being compared (EchoJEPA, EchoPrime, PanEcho, VideoMAE) uses the same probe architecture, the same LR/weight-decay sweep grid, and the same multi-view fusion strategy. The only thing that differs is the frozen encoder producing the tokens; any difference in downstream performance is therefore a difference in what the encoder learned, not in how the probe was tuned.

Any model that only ever probes one clip at a time is structurally incapable of getting this right, no matter how good its encoder is, which makes RVSP a useful way to tell "the encoder is strong" apart from "the whole system reasons correctly."

Architecture 3: Inside the multi-view attentive probe

Learning targets for the probe

Three tasks, each defined by a separate config. The probe head (a linear layer) is swapped per task; the 4-block transformer body is shared in design but trained separately for each.

Target 1: LVEF (Left Ventricular Ejection Fraction)

LVEF is a single-view task; the Apical 4-chamber view captures both the LV inflow tract and outflow tract, giving enough geometry. No cross-view reasoning is needed; the probe runs single-view, not multi-view.

Target 2: RVSP (Right Ventricular Systolic Pressure)

RVSP only uses 4 clips from 2 views:

TR velocity is the tricuspid regurgitation jet velocity (m/s), measured in the Apical view via continuous-wave Doppler. RAP is right atrial pressure (mmHg), estimated from IVC diameter and collapsibility measured in the Subcostal view. RVSP is the litmus test for whether multi-view reasoning actually works; competitors that use single-view probes cannot compute it correctly regardless of encoder quality.

Target 3: View Classification (13 classes)

Probe output is a [B, 13] softmax over view classes, trained with cross-entropy loss. This is a per-clip task, not a study-level aggregation.

Note: each of the three tasks is trained and probed independently.

Datasets used

Architectures

EchoJEPA trains two variants at different scale, EchoJEPA-G and EchoJEPA-L; the weights for EchoJEPA-L are open-sourced.

Results: EchoJEPA vs. Competing Methods

All models are evaluated with identical probes: same architecture (depth=4, 16 heads), same LR/weight-decay sweep, same multi-view fusion. The only variable is the frozen encoder.

• EchoPrime. Contrastive VLM, trained on 1M+ echo videos with report supervision.
• PanEcho. Contrastive model trained on 1M+ echo-report pairs.
• EchoMAE-L. Pixel reconstruction (VideoMAE objective), same ViT-L backbone as EchoJEPA-L; the direct apples-to-apples ablation of latent vs. pixel prediction.
• EchoJEPA-L. V-JEPA 2 latent prediction, ViT-L, trained on 525K public MIMIC-IV-Echo videos.
• EchoJEPA-G. V-JEPA 2 latent prediction, ViT-G (1.1B), trained on 18.1M proprietary echo videos.

Summary

Attention visualization

Decoding what each model's encoder actually attends to, before and after finetuning on echocardiograms, shows why latent prediction produces more anatomically grounded representations.

Latent space analysis

What this means going forward

In November 2025, LeCun announced his departure from Meta after twelve years (five as FAIR's founding director, seven as Chief AI Scientist) to co-found Advanced Machine Intelligence Labs (AMI Labs) in Paris. AMI Labs targets industrial, robotic, and healthcare applications using JEPA-based learning from video and sensor data; the fundraise signals investor confidence that world models (not LLM-style next-token prediction) are the path forward.

The JEPA family has fanned out quickly since. LLM-JEPA applies the objective to language, predicting latent token-sequence representations instead of the next token, and outperforms standard LLM training objectives. LeJEPA is a leaner, theoretically grounded reformulation of the whole framework that removes ad-hoc heuristics. ACT-JEPA conditions on actions for efficient policy representation learning in robotics. LeWorldModel adds value shaping to the JEPA world model's representation space to enable planning. VL-JEPA predicts continuous text embeddings for vision-language tasks instead of generating tokens autoregressively. US-JEPA generalizes the approach to ultrasound imaging in general, learning anatomical dependencies and tissue-texture relationships across multiple ultrasound modalities. And V-JEPA 2.1 pushes denser spatiotemporal features out of the original video model, setting new marks on Ego4D and EPIC-KITCHENS.

The JEPA family, as of this writing