Skip to content

Perov-5: alignment across seeds

The alignment model of MEIDNet learns one space for crystal structures and their properties. This page measures how well the two modalities agree in that space, and how much the answer depends on the random seed: the same training was run seven times, and every checkpoint is public.

0.954 ± 0.011cosine, 7 seeds
0.301 ± 0.034L2, 7 seeds
86.1 ± 5.0 %structure matching
18,928materials

Results · During training · Unseen materials · Do it yourself · What the runs show

What was run
model early fusion of formation enthalpy and band gap; E(n)-equivariant crystal encoder; 128-dimensional latents
training 2,200 epochs, batch 16, Adam, learning rate 1e-3; symmetric InfoNCE with temperature 0.01; contrastive weight raised linearly from 0 to 5 over the first 1,500 epochs (the curriculum)
data all 18,928 Perov-5 materials, scored on the same materials; unseen materials are measured separately
seeds 0 to 6, one run each, about 7 hours on one NVIDIA A100
script and weights reproduction/paper_alignment/ · seven checkpoints on Hugging Face

Results

seed cosine L2 structure matching, sampled element structure matching, most likely element retrieval, top 1
0 0.9563 0.2934 90.04 % 92.79 % 0.245
1 0.9585 0.2862 91.41 % 94.55 % 0.288
2 0.9373 0.3529 82.91 % 88.34 % 0.218
3 0.9592 0.2839 91.55 % 94.69 % 0.249
4 0.9654 0.2615 78.21 % 82.49 % 0.267
5 0.9586 0.2863 85.15 % 88.87 % 0.215
6 0.9400 0.3449 83.76 % 88.82 % 0.127
mean ± s.d. 0.954 ± 0.011 0.301 ± 0.034 86.1 ± 5.0 % 90.1 ± 4.3 % 0.230 ± 0.052
range 0.937–0.965 0.262–0.353 78.2–91.5 % 82.5–94.7 % 0.127–0.288
The four measures used on this page
  • Cosine and L2: how close the structure latent and the property latent of the same material are. Both latents have length one; a cosine of 1 (L2 of 0) means they coincide. They are read from the normalised encoder outputs, before the projection heads, which is where this model's alignment loss acts.
  • Structure matching: the share of materials whose crystal, decoded from the joint latent, matches the input according to pymatgen's StructureMatcher (stol 0.5, angle_tol 10, ltol 0.3). Sampled element draws the element of each site from the decoder's probabilities, as the training script's own evaluation does; most likely element takes the highest probability and is deterministic.
  • Retrieval, top 1: the share of materials whose own property vector is the nearest of all property vectors to their structure vector.

A seed fixes the result

Seeds 0, 1, 2 were trained twice, on different hardware and library versions (torch 2.6.0, pymatgen 2024.11.13, NVIDIA A100 40 GB; torch 2.8.0, pymatgen 2025.10.7, NVIDIA A100 80 GB). Every number of the two sets is identical. The differences between the rows above therefore come from the seed alone: quote a result with its seed, or as a mean over seeds.

During training

Cosine between the structure and property latents, per epoch, for the seven seeds

Cosine between the two latents during training, seven seeds05001,0001,5002,00000.250.50.751epochcosine (training batches)seed 0seed 1seed 2seed 3seed 4seed 5seed 6

The seeds differ most early in training and converge later:

epoch lowest seed highest seed spread
10 0.013 0.159 0.146
25 0.088 0.285 0.197
50 0.166 0.727 0.561
100 0.506 0.844 0.338
200 0.765 0.900 0.135
400 0.851 0.928 0.078
800 0.908 0.947 0.039
1,500 0.934 0.962 0.028
2,200 0.939 0.960 0.022

Unseen materials

The runs above score the materials the model was trained on. To measure materials it has never seen, the same training was repeated on 15,142 materials (80 %) with 3,786 (20 %) held out, with three seeds.

cosine L2 structure matching, sampled structure matching, most likely retrieval, top 1
held-out 3,786 materials 0.919 ± 0.006 0.376 ± 0.020 66.6 ± 5.9 % 69.5 ± 6.3 % 0.087–0.105
the 15,142 trained on 0.953 ± 0.008 0.305 ± 0.027 87.4 ± 4.4 % 90.8 ± 3.9 % 0.207–0.275

The alignment carries over to unseen materials with a small loss; the reconstruction of the crystal loses about twenty points. What the runs show traces that loss to one cause.

Do it yourself

Nothing to install. The seven models are compared with the other methods on the Perov-5 leaderboard, and the insights page has the figures behind every statement here.

Recompute the cosine and L2 of any seed on a CPU. The checkpoint is downloaded from Hugging Face.

pip install git+https://github.com/ABnano/MEIDNet.git
git clone https://github.com/ABnano/MEIDNet.git && cd MEIDNet
meidnet download-data                                   # Perov-5 into data/perov5/
python scripts/reproduce_alignment.py --seed 4          # prints cosine and L2
python scripts/reproduce_alignment.py --seed 4 --structure-matching   # also rebuilds every crystal: a few minutes

Expected for seed 4: cosine 0.9654, L2 0.2615, structure matching 82.49 % (most likely element). The values of every seed are in runs.csv.

The training script, the wrapper and the job files are in reproduction/paper_alignment/.

cd reproduction/paper_alignment
python -u retrain_alignment.py --seed 4 --epochs 2200           # trains, then prints cosine, L2 and structure matching
python -u retrain_alignment.py --ckpt path/to/checkpoint.pth    # evaluation only

The same seed gives the same numbers as in the table above, on any GPU.

Files