Weights & Biases¶
LaminDB can be integrated with W&B to track the training process and associate datasets & parameters with models.
# pip install lamindb torchvision lightning wandb
!lamin init --storage ./lamin-mlops
!wandb login
Show code cell output
→ connected lamindb: anonymous/lamin-mlops
• to map a local dev directory, set: ln.setup.settings.dev_dir = '.'
wandb: Currently logged in as: felix_lamin (lamin-mlops-demo) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
import lightning as pl
import lamindb as ln
from lamindb.integrations import lightning as ll
import wandb
from pathlib import Path
from torch import utils
from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
from autoencoder import LitAutoEncoder
Show code cell output
→ connected lamindb: anonymous/lamin-mlops
# define model run parameters, features, and labels so that validation passes later on
MODEL_CONFIG = {"hidden_size": 32, "bottleneck_size": 16, "batch_size": 32}
hyperparameter = ln.Feature(name="Autoencoder hyperparameter", is_type=True).save()
hyperparams = ln.Feature.from_dict(MODEL_CONFIG, type=hyperparameter)
ln.save(hyperparams)
metrics_to_annotate = ["train_loss", "val_loss", "current_epoch"]
for metric in metrics_to_annotate:
dtype = int if metric == "current_epoch" else float
ln.Feature(name=metric, dtype=dtype).save()
# create all Wandb related features like 'wandb_run_id'
ln.examples.wandb.save_wandb_features()
Show code cell output
! name 'Weights & Biases' for type ends with 's', in case you're naming with plural, consider the singular for a type name
# track this notebook/script run so that all checkpoint artifacts are associated with the source code
ln.track(params=MODEL_CONFIG, project=ln.Project(name="Wandb tutorial").save())
Show code cell output
→ created Transform('1C9yrZV09NNd0000', key='wandb.ipynb'), started new Run('snk8iVlobFFFWEjY') at 2025-10-30 07:58:46 UTC
→ params: hidden_size=32, bottleneck_size=16, batch_size=32
→ notebook imports: autoencoder lamindb==1.15a1 lightning==2.5.5 torch==2.9.0 torchvision==0.24.0 wandb==0.22.3
• recommendation: to identify the notebook across renames, pass the uid: ln.track("1C9yrZV09NNd", project="Wandb tutorial", params={...})
Define a model¶
We use a basic PyTorch Lightning autoencoder as an example model.
Code of LitAutoEncoder
Simple autoencoder model¶
import torch
import lightning
from torch import optim, nn
class LitAutoEncoder(lightning.LightningModule):
def __init__(self, hidden_size: int, bottleneck_size: int) -> None:
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(28 * 28, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, bottleneck_size),
)
self.decoder = nn.Sequential(
nn.Linear(bottleneck_size, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, 28 * 28),
)
self.save_hyperparameters()
def training_step(
self, batch: tuple[torch.Tensor, torch.Tensor], batch_idx: int
) -> torch.Tensor:
x, y = batch
x = x.view(x.size(0), -1)
z = self.encoder(x)
x_hat = self.decoder(z)
loss = nn.functional.mse_loss(x_hat, x)
self.log("train_loss", loss, on_epoch=True)
return loss
def validation_step(
self, batch: tuple[torch.Tensor, torch.Tensor], batch_idx: int
) -> torch.Tensor:
x, y = batch
x = x.view(x.size(0), -1)
z = self.encoder(x)
x_hat = self.decoder(z)
loss = nn.functional.mse_loss(x_hat, x)
self.log("val_loss", loss, on_epoch=True)
return loss
def configure_optimizers(self) -> optim.Optimizer:
optimizer = optim.Adam(self.parameters(), lr=1e-3)
return optimizer
Query & download the MNIST dataset¶
We saved the MNIST dataset in a curation notebook which now shows up in the Artifact registry:
ln.Artifact.filter(kind="dataset").to_dataframe()
Show code cell output
| uid | key | description | suffix | kind | otype | size | hash | n_files | n_observations | version | is_latest | is_locked | created_at | branch_id | space_id | storage_id | run_id | schema_id | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 1 | 8wrINeHWEosqo5xJ0000 | testdata/mnist | Complete MNIST dataset directory containing tr... | dataset | None | 54950048 | amFx_vXqnUtJr0kmxxWK2Q | 4 | None | None | True | False | 2025-10-30 07:58:27.621000+00:00 | 1 | 1 | 1 | 1 | None | 1 |
Let’s get the dataset:
artifact = ln.Artifact.get(key="testdata/mnist")
artifact
Show code cell output
Artifact(uid='8wrINeHWEosqo5xJ0000', version=None, is_latest=True, key='testdata/mnist', description='Complete MNIST dataset directory containing training and test data', suffix='', kind='dataset', otype=None, size=54950048, hash='amFx_vXqnUtJr0kmxxWK2Q', n_files=4, n_observations=None, branch_id=1, space_id=1, storage_id=1, run_id=1, schema_id=None, created_by_id=1, created_at=2025-10-30 07:58:27 UTC, is_locked=False)
And download it to a local cache:
path = artifact.cache()
path
Show code cell output
PosixUPath('/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops/.lamindb/8wrINeHWEosqo5xJ')
Create a PyTorch-compatible dataset:
dataset = MNIST(path.as_posix(), transform=ToTensor())
dataset
Show code cell output
Dataset MNIST
Number of datapoints: 60000
Root location: /home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops/.lamindb/8wrINeHWEosqo5xJ
Split: Train
StandardTransform
Transform: ToTensor()
Monitor training with wandb¶
Train our example model and track the training progress with wandb.
from lightning.pytorch.loggers import WandbLogger
# create the data loader
train_dataset = MNIST(root="./data", train=True, download=True, transform=ToTensor())
val_dataset = MNIST(root="./data", train=False, download=True, transform=ToTensor())
train_loader = utils.data.DataLoader(train_dataset, batch_size=32)
val_loader = utils.data.DataLoader(val_dataset, batch_size=32)
# init model
autoencoder = LitAutoEncoder(
MODEL_CONFIG["hidden_size"], MODEL_CONFIG["bottleneck_size"]
)
# initialize the logger
wandb_logger = WandbLogger(project="lamin")
# add batch size to the wandb config
wandb_logger.experiment.config["batch_size"] = MODEL_CONFIG["batch_size"]
Show code cell output
0%| | 0.00/9.91M [00:00<?, ?B/s]
1%| | 98.3k/9.91M [00:00<00:13, 713kB/s]
4%|▎ | 360k/9.91M [00:00<00:06, 1.58MB/s]
16%|█▌ | 1.54M/9.91M [00:00<00:01, 4.50MB/s]
61%|██████ | 6.06M/9.91M [00:00<00:00, 16.6MB/s]
100%|██████████| 9.91M/9.91M [00:00<00:00, 16.3MB/s]
0%| | 0.00/28.9k [00:00<?, ?B/s]
100%|██████████| 28.9k/28.9k [00:00<00:00, 408kB/s]
0%| | 0.00/1.65M [00:00<?, ?B/s]
4%|▍ | 65.5k/1.65M [00:00<00:03, 458kB/s]
14%|█▍ | 229k/1.65M [00:00<00:01, 858kB/s]
56%|█████▌ | 918k/1.65M [00:00<00:00, 2.63MB/s]
100%|██████████| 1.65M/1.65M [00:00<00:00, 3.77MB/s]
0%| | 0.00/4.54k [00:00<?, ?B/s]
100%|██████████| 4.54k/4.54k [00:00<00:00, 19.4MB/s]
wandb: Currently logged in as: felix_lamin (lamin-mlops-demo) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.22.3
wandb: Run data is saved locally in ./wandb/run-20251030_075850-1c84fp9x
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run ghastly-cobweb-325
wandb: ⭐️ View project at https://wandb.ai/lamin-mlops-demo/lamin
wandb: 🚀 View run at https://wandb.ai/lamin-mlops-demo/lamin/runs/1c84fp9x
# Create a LaminDB LightningCallback which also (optionally) annotates checkpoints by desired metrics
wandb_logger.experiment.id
lamindb_callback = ll.Callback(
path=Path("model_checkpoints") / f"{wandb_logger.version}_last_epoch.ckpt",
key=f"testmodels/wandb/{wandb_logger.experiment.id}.ckpt",
features={
"wandb_run_id": wandb_logger.experiment.id,
"wandb_run_name": wandb_logger.experiment.name,
**{
metric: None for metric in metrics_to_annotate
}, # auto-populated through callback
},
)
# train model
trainer = pl.Trainer(
limit_train_batches=3,
max_epochs=5,
logger=wandb_logger,
callbacks=[lamindb_callback],
)
trainer.fit(
model=autoencoder, train_dataloaders=train_loader, val_dataloaders=val_loader
)
Show code cell output
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
| Name | Type | Params | Mode
-----------------------------------------------
0 | encoder | Sequential | 25.6 K | train
1 | decoder | Sequential | 26.4 K | train
-----------------------------------------------
52.1 K Trainable params
0 Non-trainable params
52.1 K Total params
0.208 Total estimated model params size (MB)
8 Modules in train mode
0 Modules in eval mode
Sanity Checking: | | 0/? [00:00<?, ?it/s]
Sanity Checking: | | 0/? [00:00<?, ?it/s]
Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]
Sanity Checking DataLoader 0: 50%|█████ | 1/2 [00:00<00:00, 79.58it/s]
Sanity Checking DataLoader 0: 100%|██████████| 2/2 [00:00<00:00, 112.89it/s]
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:433: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance.
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:433: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance.
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/lightning/pytorch/loops/fit_loop.py:310: The number of training batches (3) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
Training: | | 0/? [00:00<?, ?it/s]
Training: | | 0/? [00:00<?, ?it/s]
Epoch 0: 0%| | 0/3 [00:00<?, ?it/s]
Epoch 0: 33%|███▎ | 1/3 [00:00<00:00, 119.01it/s]
Epoch 0: 33%|███▎ | 1/3 [00:00<00:00, 111.02it/s, v_num=fp9x]
Epoch 0: 67%|██████▋ | 2/3 [00:00<00:00, 126.49it/s, v_num=fp9x]
Epoch 0: 67%|██████▋ | 2/3 [00:00<00:00, 121.96it/s, v_num=fp9x]
Epoch 0: 100%|██████████| 3/3 [00:00<00:00, 129.33it/s, v_num=fp9x]
Epoch 0: 100%|██████████| 3/3 [00:00<00:00, 126.00it/s, v_num=fp9x]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: | | 0/? [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/313 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 1/313 [00:00<00:01, 190.68it/s]
Validation DataLoader 0: 1%| | 2/313 [00:00<00:01, 165.83it/s]
Validation DataLoader 0: 1%| | 3/313 [00:00<00:01, 163.20it/s]
Validation DataLoader 0: 1%|▏ | 4/313 [00:00<00:01, 163.02it/s]
Validation DataLoader 0: 2%|▏ | 5/313 [00:00<00:01, 163.35it/s]
Validation DataLoader 0: 2%|▏ | 6/313 [00:00<00:01, 163.18it/s]
Validation DataLoader 0: 2%|▏ | 7/313 [00:00<00:01, 163.86it/s]
Validation DataLoader 0: 3%|▎ | 8/313 [00:00<00:01, 164.25it/s]
Validation DataLoader 0: 3%|▎ | 9/313 [00:00<00:01, 164.35it/s]
Validation DataLoader 0: 3%|▎ | 10/313 [00:00<00:01, 164.50it/s]
Validation DataLoader 0: 4%|▎ | 11/313 [00:00<00:01, 164.91it/s]
Validation DataLoader 0: 4%|▍ | 12/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 4%|▍ | 13/313 [00:00<00:01, 164.95it/s]
Validation DataLoader 0: 4%|▍ | 14/313 [00:00<00:01, 165.09it/s]
Validation DataLoader 0: 5%|▍ | 15/313 [00:00<00:01, 164.05it/s]
Validation DataLoader 0: 5%|▌ | 16/313 [00:00<00:01, 163.74it/s]
Validation DataLoader 0: 5%|▌ | 17/313 [00:00<00:01, 163.19it/s]
Validation DataLoader 0: 6%|▌ | 18/313 [00:00<00:01, 163.30it/s]
Validation DataLoader 0: 6%|▌ | 19/313 [00:00<00:01, 163.58it/s]
Validation DataLoader 0: 6%|▋ | 20/313 [00:00<00:01, 163.71it/s]
Validation DataLoader 0: 7%|▋ | 21/313 [00:00<00:01, 163.94it/s]
Validation DataLoader 0: 7%|▋ | 22/313 [00:00<00:01, 163.97it/s]
Validation DataLoader 0: 7%|▋ | 23/313 [00:00<00:01, 164.00it/s]
Validation DataLoader 0: 8%|▊ | 24/313 [00:00<00:01, 164.17it/s]
Validation DataLoader 0: 8%|▊ | 25/313 [00:00<00:01, 164.19it/s]
Validation DataLoader 0: 8%|▊ | 26/313 [00:00<00:01, 164.32it/s]
Validation DataLoader 0: 9%|▊ | 27/313 [00:00<00:01, 164.27it/s]
Validation DataLoader 0: 9%|▉ | 28/313 [00:00<00:01, 164.27it/s]
Validation DataLoader 0: 9%|▉ | 29/313 [00:00<00:01, 164.38it/s]
Validation DataLoader 0: 10%|▉ | 30/313 [00:00<00:01, 164.46it/s]
Validation DataLoader 0: 10%|▉ | 31/313 [00:00<00:01, 164.56it/s]
Validation DataLoader 0: 10%|█ | 32/313 [00:00<00:01, 164.71it/s]
Validation DataLoader 0: 11%|█ | 33/313 [00:00<00:01, 164.60it/s]
Validation DataLoader 0: 11%|█ | 34/313 [00:00<00:01, 164.60it/s]
Validation DataLoader 0: 11%|█ | 35/313 [00:00<00:01, 164.56it/s]
Validation DataLoader 0: 12%|█▏ | 36/313 [00:00<00:01, 164.68it/s]
Validation DataLoader 0: 12%|█▏ | 37/313 [00:00<00:01, 164.74it/s]
Validation DataLoader 0: 12%|█▏ | 38/313 [00:00<00:01, 164.82it/s]
Validation DataLoader 0: 12%|█▏ | 39/313 [00:00<00:01, 164.88it/s]
Validation DataLoader 0: 13%|█▎ | 40/313 [00:00<00:01, 164.94it/s]
Validation DataLoader 0: 13%|█▎ | 41/313 [00:00<00:01, 165.02it/s]
Validation DataLoader 0: 13%|█▎ | 42/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 14%|█▎ | 43/313 [00:00<00:01, 165.05it/s]
Validation DataLoader 0: 14%|█▍ | 44/313 [00:00<00:01, 165.20it/s]
Validation DataLoader 0: 14%|█▍ | 45/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 15%|█▍ | 46/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 15%|█▌ | 47/313 [00:00<00:01, 165.27it/s]
Validation DataLoader 0: 15%|█▌ | 48/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 16%|█▌ | 49/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 16%|█▌ | 50/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 16%|█▋ | 51/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 17%|█▋ | 52/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 17%|█▋ | 53/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 17%|█▋ | 54/313 [00:00<00:01, 165.33it/s]
Validation DataLoader 0: 18%|█▊ | 55/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 18%|█▊ | 56/313 [00:00<00:01, 164.92it/s]
Validation DataLoader 0: 18%|█▊ | 57/313 [00:00<00:01, 164.99it/s]
Validation DataLoader 0: 19%|█▊ | 58/313 [00:00<00:01, 165.06it/s]
Validation DataLoader 0: 19%|█▉ | 59/313 [00:00<00:01, 165.10it/s]
Validation DataLoader 0: 19%|█▉ | 60/313 [00:00<00:01, 165.00it/s]
Validation DataLoader 0: 19%|█▉ | 61/313 [00:00<00:01, 164.64it/s]
Validation DataLoader 0: 20%|█▉ | 62/313 [00:00<00:01, 164.59it/s]
Validation DataLoader 0: 20%|██ | 63/313 [00:00<00:01, 164.22it/s]
Validation DataLoader 0: 20%|██ | 64/313 [00:00<00:01, 163.91it/s]
Validation DataLoader 0: 21%|██ | 65/313 [00:00<00:01, 163.87it/s]
Validation DataLoader 0: 21%|██ | 66/313 [00:00<00:01, 163.95it/s]
Validation DataLoader 0: 21%|██▏ | 67/313 [00:00<00:01, 164.00it/s]
Validation DataLoader 0: 22%|██▏ | 68/313 [00:00<00:01, 163.96it/s]
Validation DataLoader 0: 22%|██▏ | 69/313 [00:00<00:01, 163.99it/s]
Validation DataLoader 0: 22%|██▏ | 70/313 [00:00<00:01, 163.96it/s]
Validation DataLoader 0: 23%|██▎ | 71/313 [00:00<00:01, 163.96it/s]
Validation DataLoader 0: 23%|██▎ | 72/313 [00:00<00:01, 163.99it/s]
Validation DataLoader 0: 23%|██▎ | 73/313 [00:00<00:01, 163.98it/s]
Validation DataLoader 0: 24%|██▎ | 74/313 [00:00<00:01, 164.03it/s]
Validation DataLoader 0: 24%|██▍ | 75/313 [00:00<00:01, 163.90it/s]
Validation DataLoader 0: 24%|██▍ | 76/313 [00:00<00:01, 163.91it/s]
Validation DataLoader 0: 25%|██▍ | 77/313 [00:00<00:01, 163.90it/s]
Validation DataLoader 0: 25%|██▍ | 78/313 [00:00<00:01, 163.89it/s]
Validation DataLoader 0: 25%|██▌ | 79/313 [00:00<00:01, 163.90it/s]
Validation DataLoader 0: 26%|██▌ | 80/313 [00:00<00:01, 163.64it/s]
Validation DataLoader 0: 26%|██▌ | 81/313 [00:00<00:01, 163.00it/s]
Validation DataLoader 0: 26%|██▌ | 82/313 [00:00<00:01, 162.50it/s]
Validation DataLoader 0: 27%|██▋ | 83/313 [00:00<00:01, 162.29it/s]
Validation DataLoader 0: 27%|██▋ | 84/313 [00:00<00:01, 162.30it/s]
Validation DataLoader 0: 27%|██▋ | 85/313 [00:00<00:01, 162.15it/s]
Validation DataLoader 0: 27%|██▋ | 86/313 [00:00<00:01, 162.20it/s]
Validation DataLoader 0: 28%|██▊ | 87/313 [00:00<00:01, 162.21it/s]
Validation DataLoader 0: 28%|██▊ | 88/313 [00:00<00:01, 162.23it/s]
Validation DataLoader 0: 28%|██▊ | 89/313 [00:00<00:01, 162.22it/s]
Validation DataLoader 0: 29%|██▉ | 90/313 [00:00<00:01, 162.25it/s]
Validation DataLoader 0: 29%|██▉ | 91/313 [00:00<00:01, 162.24it/s]
Validation DataLoader 0: 29%|██▉ | 92/313 [00:00<00:01, 162.25it/s]
Validation DataLoader 0: 30%|██▉ | 93/313 [00:00<00:01, 162.29it/s]
Validation DataLoader 0: 30%|███ | 94/313 [00:00<00:01, 162.32it/s]
Validation DataLoader 0: 30%|███ | 95/313 [00:00<00:01, 162.35it/s]
Validation DataLoader 0: 31%|███ | 96/313 [00:00<00:01, 162.42it/s]
Validation DataLoader 0: 31%|███ | 97/313 [00:00<00:01, 162.45it/s]
Validation DataLoader 0: 31%|███▏ | 98/313 [00:00<00:01, 162.51it/s]
Validation DataLoader 0: 32%|███▏ | 99/313 [00:00<00:01, 162.56it/s]
Validation DataLoader 0: 32%|███▏ | 100/313 [00:00<00:01, 162.63it/s]
Validation DataLoader 0: 32%|███▏ | 101/313 [00:00<00:01, 162.69it/s]
Validation DataLoader 0: 33%|███▎ | 102/313 [00:00<00:01, 162.73it/s]
Validation DataLoader 0: 33%|███▎ | 103/313 [00:00<00:01, 162.76it/s]
Validation DataLoader 0: 33%|███▎ | 104/313 [00:00<00:01, 162.78it/s]
Validation DataLoader 0: 34%|███▎ | 105/313 [00:00<00:01, 162.76it/s]
Validation DataLoader 0: 34%|███▍ | 106/313 [00:00<00:01, 162.84it/s]
Validation DataLoader 0: 34%|███▍ | 107/313 [00:00<00:01, 162.88it/s]
Validation DataLoader 0: 35%|███▍ | 108/313 [00:00<00:01, 162.91it/s]
Validation DataLoader 0: 35%|███▍ | 109/313 [00:00<00:01, 162.95it/s]
Validation DataLoader 0: 35%|███▌ | 110/313 [00:00<00:01, 162.94it/s]
Validation DataLoader 0: 35%|███▌ | 111/313 [00:00<00:01, 163.00it/s]
Validation DataLoader 0: 36%|███▌ | 112/313 [00:00<00:01, 163.04it/s]
Validation DataLoader 0: 36%|███▌ | 113/313 [00:00<00:01, 163.06it/s]
Validation DataLoader 0: 36%|███▋ | 114/313 [00:00<00:01, 163.07it/s]
Validation DataLoader 0: 37%|███▋ | 115/313 [00:00<00:01, 163.13it/s]
Validation DataLoader 0: 37%|███▋ | 116/313 [00:00<00:01, 163.17it/s]
Validation DataLoader 0: 37%|███▋ | 117/313 [00:00<00:01, 163.19it/s]
Validation DataLoader 0: 38%|███▊ | 118/313 [00:00<00:01, 163.21it/s]
Validation DataLoader 0: 38%|███▊ | 119/313 [00:00<00:01, 163.23it/s]
Validation DataLoader 0: 38%|███▊ | 120/313 [00:00<00:01, 163.23it/s]
Validation DataLoader 0: 39%|███▊ | 121/313 [00:00<00:01, 163.25it/s]
Validation DataLoader 0: 39%|███▉ | 122/313 [00:00<00:01, 163.25it/s]
Validation DataLoader 0: 39%|███▉ | 123/313 [00:00<00:01, 163.29it/s]
Validation DataLoader 0: 40%|███▉ | 124/313 [00:00<00:01, 163.32it/s]
Validation DataLoader 0: 40%|███▉ | 125/313 [00:00<00:01, 163.22it/s]
Validation DataLoader 0: 40%|████ | 126/313 [00:00<00:01, 163.24it/s]
Validation DataLoader 0: 41%|████ | 127/313 [00:00<00:01, 163.25it/s]
Validation DataLoader 0: 41%|████ | 128/313 [00:00<00:01, 163.27it/s]
Validation DataLoader 0: 41%|████ | 129/313 [00:00<00:01, 163.28it/s]
Validation DataLoader 0: 42%|████▏ | 130/313 [00:00<00:01, 163.31it/s]
Validation DataLoader 0: 42%|████▏ | 131/313 [00:00<00:01, 163.31it/s]
Validation DataLoader 0: 42%|████▏ | 132/313 [00:00<00:01, 163.34it/s]
Validation DataLoader 0: 42%|████▏ | 133/313 [00:00<00:01, 163.36it/s]
Validation DataLoader 0: 43%|████▎ | 134/313 [00:00<00:01, 163.39it/s]
Validation DataLoader 0: 43%|████▎ | 135/313 [00:00<00:01, 163.40it/s]
Validation DataLoader 0: 43%|████▎ | 136/313 [00:00<00:01, 163.44it/s]
Validation DataLoader 0: 44%|████▍ | 137/313 [00:00<00:01, 163.48it/s]
Validation DataLoader 0: 44%|████▍ | 138/313 [00:00<00:01, 163.51it/s]
Validation DataLoader 0: 44%|████▍ | 139/313 [00:00<00:01, 163.53it/s]
Validation DataLoader 0: 45%|████▍ | 140/313 [00:00<00:01, 163.54it/s]
Validation DataLoader 0: 45%|████▌ | 141/313 [00:00<00:01, 163.57it/s]
Validation DataLoader 0: 45%|████▌ | 142/313 [00:00<00:01, 163.59it/s]
Validation DataLoader 0: 46%|████▌ | 143/313 [00:00<00:01, 163.62it/s]
Validation DataLoader 0: 46%|████▌ | 144/313 [00:00<00:01, 163.65it/s]
Validation DataLoader 0: 46%|████▋ | 145/313 [00:00<00:01, 163.67it/s]
Validation DataLoader 0: 47%|████▋ | 146/313 [00:00<00:01, 163.69it/s]
Validation DataLoader 0: 47%|████▋ | 147/313 [00:00<00:01, 163.71it/s]
Validation DataLoader 0: 47%|████▋ | 148/313 [00:00<00:01, 163.71it/s]
Validation DataLoader 0: 48%|████▊ | 149/313 [00:00<00:01, 163.73it/s]
Validation DataLoader 0: 48%|████▊ | 150/313 [00:00<00:00, 163.77it/s]
Validation DataLoader 0: 48%|████▊ | 151/313 [00:00<00:00, 163.79it/s]
Validation DataLoader 0: 49%|████▊ | 152/313 [00:00<00:00, 163.81it/s]
Validation DataLoader 0: 49%|████▉ | 153/313 [00:00<00:00, 163.83it/s]
Validation DataLoader 0: 49%|████▉ | 154/313 [00:00<00:00, 163.83it/s]
Validation DataLoader 0: 50%|████▉ | 155/313 [00:00<00:00, 163.85it/s]
Validation DataLoader 0: 50%|████▉ | 156/313 [00:00<00:00, 163.86it/s]
Validation DataLoader 0: 50%|█████ | 157/313 [00:00<00:00, 163.88it/s]
Validation DataLoader 0: 50%|█████ | 158/313 [00:00<00:00, 163.88it/s]
Validation DataLoader 0: 51%|█████ | 159/313 [00:00<00:00, 163.88it/s]
Validation DataLoader 0: 51%|█████ | 160/313 [00:00<00:00, 163.89it/s]
Validation DataLoader 0: 51%|█████▏ | 161/313 [00:00<00:00, 163.89it/s]
Validation DataLoader 0: 52%|█████▏ | 162/313 [00:00<00:00, 163.88it/s]
Validation DataLoader 0: 52%|█████▏ | 163/313 [00:00<00:00, 163.89it/s]
Validation DataLoader 0: 52%|█████▏ | 164/313 [00:01<00:00, 163.88it/s]
Validation DataLoader 0: 53%|█████▎ | 165/313 [00:01<00:00, 163.88it/s]
Validation DataLoader 0: 53%|█████▎ | 166/313 [00:01<00:00, 163.88it/s]
Validation DataLoader 0: 53%|█████▎ | 167/313 [00:01<00:00, 163.88it/s]
Validation DataLoader 0: 54%|█████▎ | 168/313 [00:01<00:00, 163.88it/s]
Validation DataLoader 0: 54%|█████▍ | 169/313 [00:01<00:00, 163.89it/s]
Validation DataLoader 0: 54%|█████▍ | 170/313 [00:01<00:00, 163.92it/s]
Validation DataLoader 0: 55%|█████▍ | 171/313 [00:01<00:00, 163.93it/s]
Validation DataLoader 0: 55%|█████▍ | 172/313 [00:01<00:00, 163.94it/s]
Validation DataLoader 0: 55%|█████▌ | 173/313 [00:01<00:00, 163.87it/s]
Validation DataLoader 0: 56%|█████▌ | 174/313 [00:01<00:00, 163.92it/s]
Validation DataLoader 0: 56%|█████▌ | 175/313 [00:01<00:00, 163.93it/s]
Validation DataLoader 0: 56%|█████▌ | 176/313 [00:01<00:00, 163.96it/s]
Validation DataLoader 0: 57%|█████▋ | 177/313 [00:01<00:00, 163.98it/s]
Validation DataLoader 0: 57%|█████▋ | 178/313 [00:01<00:00, 163.99it/s]
Validation DataLoader 0: 57%|█████▋ | 179/313 [00:01<00:00, 164.01it/s]
Validation DataLoader 0: 58%|█████▊ | 180/313 [00:01<00:00, 164.02it/s]
Validation DataLoader 0: 58%|█████▊ | 181/313 [00:01<00:00, 164.04it/s]
Validation DataLoader 0: 58%|█████▊ | 182/313 [00:01<00:00, 164.07it/s]
Validation DataLoader 0: 58%|█████▊ | 183/313 [00:01<00:00, 164.09it/s]
Validation DataLoader 0: 59%|█████▉ | 184/313 [00:01<00:00, 164.11it/s]
Validation DataLoader 0: 59%|█████▉ | 185/313 [00:01<00:00, 164.14it/s]
Validation DataLoader 0: 59%|█████▉ | 186/313 [00:01<00:00, 164.17it/s]
Validation DataLoader 0: 60%|█████▉ | 187/313 [00:01<00:00, 164.18it/s]
Validation DataLoader 0: 60%|██████ | 188/313 [00:01<00:00, 164.20it/s]
Validation DataLoader 0: 60%|██████ | 189/313 [00:01<00:00, 164.20it/s]
Validation DataLoader 0: 61%|██████ | 190/313 [00:01<00:00, 164.22it/s]
Validation DataLoader 0: 61%|██████ | 191/313 [00:01<00:00, 164.22it/s]
Validation DataLoader 0: 61%|██████▏ | 192/313 [00:01<00:00, 164.23it/s]
Validation DataLoader 0: 62%|██████▏ | 193/313 [00:01<00:00, 164.25it/s]
Validation DataLoader 0: 62%|██████▏ | 194/313 [00:01<00:00, 164.27it/s]
Validation DataLoader 0: 62%|██████▏ | 195/313 [00:01<00:00, 164.27it/s]
Validation DataLoader 0: 63%|██████▎ | 196/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 63%|██████▎ | 197/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 63%|██████▎ | 198/313 [00:01<00:00, 164.29it/s]
Validation DataLoader 0: 64%|██████▎ | 199/313 [00:01<00:00, 164.29it/s]
Validation DataLoader 0: 64%|██████▍ | 200/313 [00:01<00:00, 164.30it/s]
Validation DataLoader 0: 64%|██████▍ | 201/313 [00:01<00:00, 164.30it/s]
Validation DataLoader 0: 65%|██████▍ | 202/313 [00:01<00:00, 164.29it/s]
Validation DataLoader 0: 65%|██████▍ | 203/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 65%|██████▌ | 204/313 [00:01<00:00, 164.27it/s]
Validation DataLoader 0: 65%|██████▌ | 205/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 66%|██████▌ | 206/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 66%|██████▌ | 207/313 [00:01<00:00, 164.29it/s]
Validation DataLoader 0: 66%|██████▋ | 208/313 [00:01<00:00, 164.28it/s]
Validation DataLoader 0: 67%|██████▋ | 209/313 [00:01<00:00, 164.30it/s]
Validation DataLoader 0: 67%|██████▋ | 210/313 [00:01<00:00, 164.32it/s]
Validation DataLoader 0: 67%|██████▋ | 211/313 [00:01<00:00, 164.32it/s]
Validation DataLoader 0: 68%|██████▊ | 212/313 [00:01<00:00, 164.32it/s]
Validation DataLoader 0: 68%|██████▊ | 213/313 [00:01<00:00, 164.33it/s]
Validation DataLoader 0: 68%|██████▊ | 214/313 [00:01<00:00, 164.34it/s]
Validation DataLoader 0: 69%|██████▊ | 215/313 [00:01<00:00, 164.36it/s]
Validation DataLoader 0: 69%|██████▉ | 216/313 [00:01<00:00, 164.37it/s]
Validation DataLoader 0: 69%|██████▉ | 217/313 [00:01<00:00, 164.37it/s]
Validation DataLoader 0: 70%|██████▉ | 218/313 [00:01<00:00, 164.38it/s]
Validation DataLoader 0: 70%|██████▉ | 219/313 [00:01<00:00, 164.40it/s]
Validation DataLoader 0: 70%|███████ | 220/313 [00:01<00:00, 164.42it/s]
Validation DataLoader 0: 71%|███████ | 221/313 [00:01<00:00, 164.42it/s]
Validation DataLoader 0: 71%|███████ | 222/313 [00:01<00:00, 164.44it/s]
Validation DataLoader 0: 71%|███████ | 223/313 [00:01<00:00, 164.45it/s]
Validation DataLoader 0: 72%|███████▏ | 224/313 [00:01<00:00, 164.45it/s]
Validation DataLoader 0: 72%|███████▏ | 225/313 [00:01<00:00, 164.46it/s]
Validation DataLoader 0: 72%|███████▏ | 226/313 [00:01<00:00, 164.48it/s]
Validation DataLoader 0: 73%|███████▎ | 227/313 [00:01<00:00, 164.48it/s]
Validation DataLoader 0: 73%|███████▎ | 228/313 [00:01<00:00, 164.50it/s]
Validation DataLoader 0: 73%|███████▎ | 229/313 [00:01<00:00, 164.50it/s]
Validation DataLoader 0: 73%|███████▎ | 230/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 74%|███████▍ | 231/313 [00:01<00:00, 164.52it/s]
Validation DataLoader 0: 74%|███████▍ | 232/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 74%|███████▍ | 233/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 75%|███████▍ | 234/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 75%|███████▌ | 235/313 [00:01<00:00, 164.55it/s]
Validation DataLoader 0: 75%|███████▌ | 236/313 [00:01<00:00, 164.55it/s]
Validation DataLoader 0: 76%|███████▌ | 237/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 76%|███████▌ | 238/313 [00:01<00:00, 164.57it/s]
Validation DataLoader 0: 76%|███████▋ | 239/313 [00:01<00:00, 164.57it/s]
Validation DataLoader 0: 77%|███████▋ | 240/313 [00:01<00:00, 164.57it/s]
Validation DataLoader 0: 77%|███████▋ | 241/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 77%|███████▋ | 242/313 [00:01<00:00, 164.55it/s]
Validation DataLoader 0: 78%|███████▊ | 243/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 78%|███████▊ | 244/313 [00:01<00:00, 164.47it/s]
Validation DataLoader 0: 78%|███████▊ | 245/313 [00:01<00:00, 164.49it/s]
Validation DataLoader 0: 79%|███████▊ | 246/313 [00:01<00:00, 164.45it/s]
Validation DataLoader 0: 79%|███████▉ | 247/313 [00:01<00:00, 164.46it/s]
Validation DataLoader 0: 79%|███████▉ | 248/313 [00:01<00:00, 164.47it/s]
Validation DataLoader 0: 80%|███████▉ | 249/313 [00:01<00:00, 164.49it/s]
Validation DataLoader 0: 80%|███████▉ | 250/313 [00:01<00:00, 164.50it/s]
Validation DataLoader 0: 80%|████████ | 251/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 81%|████████ | 252/313 [00:01<00:00, 164.52it/s]
Validation DataLoader 0: 81%|████████ | 253/313 [00:01<00:00, 164.55it/s]
Validation DataLoader 0: 81%|████████ | 254/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 81%|████████▏ | 255/313 [00:01<00:00, 164.50it/s]
Validation DataLoader 0: 82%|████████▏ | 256/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 82%|████████▏ | 257/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 82%|████████▏ | 258/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 83%|████████▎ | 259/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 83%|████████▎ | 260/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 83%|████████▎ | 261/313 [00:01<00:00, 164.57it/s]
Validation DataLoader 0: 84%|████████▎ | 262/313 [00:01<00:00, 164.58it/s]
Validation DataLoader 0: 84%|████████▍ | 263/313 [00:01<00:00, 164.58it/s]
Validation DataLoader 0: 84%|████████▍ | 264/313 [00:01<00:00, 164.59it/s]
Validation DataLoader 0: 85%|████████▍ | 265/313 [00:01<00:00, 164.60it/s]
Validation DataLoader 0: 85%|████████▍ | 266/313 [00:01<00:00, 164.61it/s]
Validation DataLoader 0: 85%|████████▌ | 267/313 [00:01<00:00, 164.63it/s]
Validation DataLoader 0: 86%|████████▌ | 268/313 [00:01<00:00, 164.64it/s]
Validation DataLoader 0: 86%|████████▌ | 269/313 [00:01<00:00, 164.64it/s]
Validation DataLoader 0: 86%|████████▋ | 270/313 [00:01<00:00, 164.66it/s]
Validation DataLoader 0: 87%|████████▋ | 271/313 [00:01<00:00, 164.66it/s]
Validation DataLoader 0: 87%|████████▋ | 272/313 [00:01<00:00, 164.67it/s]
Validation DataLoader 0: 87%|████████▋ | 273/313 [00:01<00:00, 164.67it/s]
Validation DataLoader 0: 88%|████████▊ | 274/313 [00:01<00:00, 164.67it/s]
Validation DataLoader 0: 88%|████████▊ | 275/313 [00:01<00:00, 164.68it/s]
Validation DataLoader 0: 88%|████████▊ | 276/313 [00:01<00:00, 164.69it/s]
Validation DataLoader 0: 88%|████████▊ | 277/313 [00:01<00:00, 164.69it/s]
Validation DataLoader 0: 89%|████████▉ | 278/313 [00:01<00:00, 164.69it/s]
Validation DataLoader 0: 89%|████████▉ | 279/313 [00:01<00:00, 164.69it/s]
Validation DataLoader 0: 89%|████████▉ | 280/313 [00:01<00:00, 164.70it/s]
Validation DataLoader 0: 90%|████████▉ | 281/313 [00:01<00:00, 164.70it/s]
Validation DataLoader 0: 90%|█████████ | 282/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 90%|█████████ | 283/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 91%|█████████ | 284/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 91%|█████████ | 285/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 91%|█████████▏| 286/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 92%|█████████▏| 287/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 92%|█████████▏| 288/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 92%|█████████▏| 289/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 93%|█████████▎| 290/313 [00:01<00:00, 164.74it/s]
Validation DataLoader 0: 93%|█████████▎| 291/313 [00:01<00:00, 164.71it/s]
Validation DataLoader 0: 93%|█████████▎| 292/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 94%|█████████▎| 293/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 94%|█████████▍| 294/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 94%|█████████▍| 295/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 95%|█████████▍| 296/313 [00:01<00:00, 164.72it/s]
Validation DataLoader 0: 95%|█████████▍| 297/313 [00:01<00:00, 164.74it/s]
Validation DataLoader 0: 95%|█████████▌| 298/313 [00:01<00:00, 164.74it/s]
Validation DataLoader 0: 96%|█████████▌| 299/313 [00:01<00:00, 164.76it/s]
Validation DataLoader 0: 96%|█████████▌| 300/313 [00:01<00:00, 164.76it/s]
Validation DataLoader 0: 96%|█████████▌| 301/313 [00:01<00:00, 164.77it/s]
Validation DataLoader 0: 96%|█████████▋| 302/313 [00:01<00:00, 164.77it/s]
Validation DataLoader 0: 97%|█████████▋| 303/313 [00:01<00:00, 164.79it/s]
Validation DataLoader 0: 97%|█████████▋| 304/313 [00:01<00:00, 164.79it/s]
Validation DataLoader 0: 97%|█████████▋| 305/313 [00:01<00:00, 164.80it/s]
Validation DataLoader 0: 98%|█████████▊| 306/313 [00:01<00:00, 164.80it/s]
Validation DataLoader 0: 98%|█████████▊| 307/313 [00:01<00:00, 164.80it/s]
Validation DataLoader 0: 98%|█████████▊| 308/313 [00:01<00:00, 164.80it/s]
Validation DataLoader 0: 99%|█████████▊| 309/313 [00:01<00:00, 164.81it/s]
Validation DataLoader 0: 99%|█████████▉| 310/313 [00:01<00:00, 164.82it/s]
Validation DataLoader 0: 99%|█████████▉| 311/313 [00:01<00:00, 164.82it/s]
Validation DataLoader 0: 100%|█████████▉| 312/313 [00:01<00:00, 164.82it/s]
Validation DataLoader 0: 100%|██████████| 313/313 [00:01<00:00, 164.96it/s]
Epoch 0: 100%|██████████| 3/3 [00:01<00:00, 1.55it/s, v_num=fp9x]
! calling anonymously, will miss private instances
Epoch 0: 100%|██████████| 3/3 [00:02<00:00, 1.32it/s, v_num=fp9x]
Epoch 0: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 1: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 1: 33%|███▎ | 1/3 [00:00<00:00, 127.16it/s, v_num=fp9x]
Epoch 1: 33%|███▎ | 1/3 [00:00<00:00, 118.34it/s, v_num=fp9x]
Epoch 1: 67%|██████▋ | 2/3 [00:00<00:00, 132.89it/s, v_num=fp9x]
Epoch 1: 67%|██████▋ | 2/3 [00:00<00:00, 128.59it/s, v_num=fp9x]
Epoch 1: 100%|██████████| 3/3 [00:00<00:00, 134.98it/s, v_num=fp9x]
Epoch 1: 100%|██████████| 3/3 [00:00<00:00, 130.99it/s, v_num=fp9x]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: | | 0/? [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/313 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 1/313 [00:00<00:01, 229.94it/s]
Validation DataLoader 0: 1%| | 2/313 [00:00<00:01, 194.33it/s]
Validation DataLoader 0: 1%| | 3/313 [00:00<00:01, 183.36it/s]
Validation DataLoader 0: 1%|▏ | 4/313 [00:00<00:01, 179.22it/s]
Validation DataLoader 0: 2%|▏ | 5/313 [00:00<00:01, 176.41it/s]
Validation DataLoader 0: 2%|▏ | 6/313 [00:00<00:01, 174.57it/s]
Validation DataLoader 0: 2%|▏ | 7/313 [00:00<00:01, 173.75it/s]
Validation DataLoader 0: 3%|▎ | 8/313 [00:00<00:01, 172.90it/s]
Validation DataLoader 0: 3%|▎ | 9/313 [00:00<00:01, 172.61it/s]
Validation DataLoader 0: 3%|▎ | 10/313 [00:00<00:01, 172.22it/s]
Validation DataLoader 0: 4%|▎ | 11/313 [00:00<00:01, 171.94it/s]
Validation DataLoader 0: 4%|▍ | 12/313 [00:00<00:01, 171.61it/s]
Validation DataLoader 0: 4%|▍ | 13/313 [00:00<00:01, 171.20it/s]
Validation DataLoader 0: 4%|▍ | 14/313 [00:00<00:01, 170.98it/s]
Validation DataLoader 0: 5%|▍ | 15/313 [00:00<00:01, 170.70it/s]
Validation DataLoader 0: 5%|▌ | 16/313 [00:00<00:01, 170.45it/s]
Validation DataLoader 0: 5%|▌ | 17/313 [00:00<00:01, 170.37it/s]
Validation DataLoader 0: 6%|▌ | 18/313 [00:00<00:01, 170.24it/s]
Validation DataLoader 0: 6%|▌ | 19/313 [00:00<00:01, 170.12it/s]
Validation DataLoader 0: 6%|▋ | 20/313 [00:00<00:01, 170.04it/s]
Validation DataLoader 0: 7%|▋ | 21/313 [00:00<00:01, 170.04it/s]
Validation DataLoader 0: 7%|▋ | 22/313 [00:00<00:01, 169.88it/s]
Validation DataLoader 0: 7%|▋ | 23/313 [00:00<00:01, 169.75it/s]
Validation DataLoader 0: 8%|▊ | 24/313 [00:00<00:01, 169.71it/s]
Validation DataLoader 0: 8%|▊ | 25/313 [00:00<00:01, 169.64it/s]
Validation DataLoader 0: 8%|▊ | 26/313 [00:00<00:01, 169.60it/s]
Validation DataLoader 0: 9%|▊ | 27/313 [00:00<00:01, 169.57it/s]
Validation DataLoader 0: 9%|▉ | 28/313 [00:00<00:01, 169.49it/s]
Validation DataLoader 0: 9%|▉ | 29/313 [00:00<00:01, 169.49it/s]
Validation DataLoader 0: 10%|▉ | 30/313 [00:00<00:01, 169.37it/s]
Validation DataLoader 0: 10%|▉ | 31/313 [00:00<00:01, 169.05it/s]
Validation DataLoader 0: 10%|█ | 32/313 [00:00<00:01, 168.89it/s]
Validation DataLoader 0: 11%|█ | 33/313 [00:00<00:01, 168.78it/s]
Validation DataLoader 0: 11%|█ | 34/313 [00:00<00:01, 168.68it/s]
Validation DataLoader 0: 11%|█ | 35/313 [00:00<00:01, 168.50it/s]
Validation DataLoader 0: 12%|█▏ | 36/313 [00:00<00:01, 168.38it/s]
Validation DataLoader 0: 12%|█▏ | 37/313 [00:00<00:01, 168.29it/s]
Validation DataLoader 0: 12%|█▏ | 38/313 [00:00<00:01, 168.08it/s]
Validation DataLoader 0: 12%|█▏ | 39/313 [00:00<00:01, 167.72it/s]
Validation DataLoader 0: 13%|█▎ | 40/313 [00:00<00:01, 167.64it/s]
Validation DataLoader 0: 13%|█▎ | 41/313 [00:00<00:01, 167.56it/s]
Validation DataLoader 0: 13%|█▎ | 42/313 [00:00<00:01, 167.54it/s]
Validation DataLoader 0: 14%|█▎ | 43/313 [00:00<00:01, 167.53it/s]
Validation DataLoader 0: 14%|█▍ | 44/313 [00:00<00:01, 167.44it/s]
Validation DataLoader 0: 14%|█▍ | 45/313 [00:00<00:01, 167.39it/s]
Validation DataLoader 0: 15%|█▍ | 46/313 [00:00<00:01, 167.37it/s]
Validation DataLoader 0: 15%|█▌ | 47/313 [00:00<00:01, 167.38it/s]
Validation DataLoader 0: 15%|█▌ | 48/313 [00:00<00:01, 167.31it/s]
Validation DataLoader 0: 16%|█▌ | 49/313 [00:00<00:01, 167.29it/s]
Validation DataLoader 0: 16%|█▌ | 50/313 [00:00<00:01, 167.30it/s]
Validation DataLoader 0: 16%|█▋ | 51/313 [00:00<00:01, 167.28it/s]
Validation DataLoader 0: 17%|█▋ | 52/313 [00:00<00:01, 167.24it/s]
Validation DataLoader 0: 17%|█▋ | 53/313 [00:00<00:01, 167.27it/s]
Validation DataLoader 0: 17%|█▋ | 54/313 [00:00<00:01, 167.29it/s]
Validation DataLoader 0: 18%|█▊ | 55/313 [00:00<00:01, 167.22it/s]
Validation DataLoader 0: 18%|█▊ | 56/313 [00:00<00:01, 167.26it/s]
Validation DataLoader 0: 18%|█▊ | 57/313 [00:00<00:01, 167.22it/s]
Validation DataLoader 0: 19%|█▊ | 58/313 [00:00<00:01, 167.19it/s]
Validation DataLoader 0: 19%|█▉ | 59/313 [00:00<00:01, 167.19it/s]
Validation DataLoader 0: 19%|█▉ | 60/313 [00:00<00:01, 167.18it/s]
Validation DataLoader 0: 19%|█▉ | 61/313 [00:00<00:01, 167.15it/s]
Validation DataLoader 0: 20%|█▉ | 62/313 [00:00<00:01, 167.18it/s]
Validation DataLoader 0: 20%|██ | 63/313 [00:00<00:01, 167.18it/s]
Validation DataLoader 0: 20%|██ | 64/313 [00:00<00:01, 167.21it/s]
Validation DataLoader 0: 21%|██ | 65/313 [00:00<00:01, 167.13it/s]
Validation DataLoader 0: 21%|██ | 66/313 [00:00<00:01, 167.14it/s]
Validation DataLoader 0: 21%|██▏ | 67/313 [00:00<00:01, 167.14it/s]
Validation DataLoader 0: 22%|██▏ | 68/313 [00:00<00:01, 167.14it/s]
Validation DataLoader 0: 22%|██▏ | 69/313 [00:00<00:01, 167.14it/s]
Validation DataLoader 0: 22%|██▏ | 70/313 [00:00<00:01, 167.15it/s]
Validation DataLoader 0: 23%|██▎ | 71/313 [00:00<00:01, 167.15it/s]
Validation DataLoader 0: 23%|██▎ | 72/313 [00:00<00:01, 167.14it/s]
Validation DataLoader 0: 23%|██▎ | 73/313 [00:00<00:01, 167.08it/s]
Validation DataLoader 0: 24%|██▎ | 74/313 [00:00<00:01, 167.06it/s]
Validation DataLoader 0: 24%|██▍ | 75/313 [00:00<00:01, 167.01it/s]
Validation DataLoader 0: 24%|██▍ | 76/313 [00:00<00:01, 166.94it/s]
Validation DataLoader 0: 25%|██▍ | 77/313 [00:00<00:01, 166.89it/s]
Validation DataLoader 0: 25%|██▍ | 78/313 [00:00<00:01, 166.86it/s]
Validation DataLoader 0: 25%|██▌ | 79/313 [00:00<00:01, 166.87it/s]
Validation DataLoader 0: 26%|██▌ | 80/313 [00:00<00:01, 166.83it/s]
Validation DataLoader 0: 26%|██▌ | 81/313 [00:00<00:01, 166.81it/s]
Validation DataLoader 0: 26%|██▌ | 82/313 [00:00<00:01, 166.52it/s]
Validation DataLoader 0: 27%|██▋ | 83/313 [00:00<00:01, 166.15it/s]
Validation DataLoader 0: 27%|██▋ | 84/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 27%|██▋ | 85/313 [00:00<00:01, 165.68it/s]
Validation DataLoader 0: 27%|██▋ | 86/313 [00:00<00:01, 165.60it/s]
Validation DataLoader 0: 28%|██▊ | 87/313 [00:00<00:01, 165.62it/s]
Validation DataLoader 0: 28%|██▊ | 88/313 [00:00<00:01, 165.61it/s]
Validation DataLoader 0: 28%|██▊ | 89/313 [00:00<00:01, 165.58it/s]
Validation DataLoader 0: 29%|██▉ | 90/313 [00:00<00:01, 165.59it/s]
Validation DataLoader 0: 29%|██▉ | 91/313 [00:00<00:01, 165.56it/s]
Validation DataLoader 0: 29%|██▉ | 92/313 [00:00<00:01, 165.50it/s]
Validation DataLoader 0: 30%|██▉ | 93/313 [00:00<00:01, 165.51it/s]
Validation DataLoader 0: 30%|███ | 94/313 [00:00<00:01, 165.52it/s]
Validation DataLoader 0: 30%|███ | 95/313 [00:00<00:01, 165.52it/s]
Validation DataLoader 0: 31%|███ | 96/313 [00:00<00:01, 165.50it/s]
Validation DataLoader 0: 31%|███ | 97/313 [00:00<00:01, 165.50it/s]
Validation DataLoader 0: 31%|███▏ | 98/313 [00:00<00:01, 165.50it/s]
Validation DataLoader 0: 32%|███▏ | 99/313 [00:00<00:01, 165.46it/s]
Validation DataLoader 0: 32%|███▏ | 100/313 [00:00<00:01, 165.47it/s]
Validation DataLoader 0: 32%|███▏ | 101/313 [00:00<00:01, 165.43it/s]
Validation DataLoader 0: 33%|███▎ | 102/313 [00:00<00:01, 165.41it/s]
Validation DataLoader 0: 33%|███▎ | 103/313 [00:00<00:01, 165.39it/s]
Validation DataLoader 0: 33%|███▎ | 104/313 [00:00<00:01, 165.39it/s]
Validation DataLoader 0: 34%|███▎ | 105/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 34%|███▍ | 106/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 34%|███▍ | 107/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 35%|███▍ | 108/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 35%|███▍ | 109/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 35%|███▌ | 110/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 35%|███▌ | 111/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 36%|███▌ | 112/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 36%|███▌ | 113/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 36%|███▋ | 114/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 37%|███▋ | 115/313 [00:00<00:01, 165.33it/s]
Validation DataLoader 0: 37%|███▋ | 116/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 37%|███▋ | 117/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 38%|███▊ | 118/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 38%|███▊ | 119/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 38%|███▊ | 120/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 39%|███▊ | 121/313 [00:00<00:01, 165.20it/s]
Validation DataLoader 0: 39%|███▉ | 122/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 39%|███▉ | 123/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 40%|███▉ | 124/313 [00:00<00:01, 165.18it/s]
Validation DataLoader 0: 40%|███▉ | 125/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 40%|████ | 126/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 41%|████ | 127/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 41%|████ | 128/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 41%|████ | 129/313 [00:00<00:01, 165.27it/s]
Validation DataLoader 0: 42%|████▏ | 130/313 [00:00<00:01, 165.31it/s]
Validation DataLoader 0: 42%|████▏ | 131/313 [00:00<00:01, 165.33it/s]
Validation DataLoader 0: 42%|████▏ | 132/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 42%|████▏ | 133/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 43%|████▎ | 134/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 43%|████▎ | 135/313 [00:00<00:01, 165.39it/s]
Validation DataLoader 0: 43%|████▎ | 136/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 44%|████▍ | 137/313 [00:00<00:01, 165.40it/s]
Validation DataLoader 0: 44%|████▍ | 138/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 44%|████▍ | 139/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 45%|████▍ | 140/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 45%|████▌ | 141/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 45%|████▌ | 142/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 46%|████▌ | 143/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 46%|████▌ | 144/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 46%|████▋ | 145/313 [00:00<00:01, 165.39it/s]
Validation DataLoader 0: 47%|████▋ | 146/313 [00:00<00:01, 165.40it/s]
Validation DataLoader 0: 47%|████▋ | 147/313 [00:00<00:01, 165.40it/s]
Validation DataLoader 0: 47%|████▋ | 148/313 [00:00<00:00, 165.41it/s]
Validation DataLoader 0: 48%|████▊ | 149/313 [00:00<00:00, 165.43it/s]
Validation DataLoader 0: 48%|████▊ | 150/313 [00:00<00:00, 165.44it/s]
Validation DataLoader 0: 48%|████▊ | 151/313 [00:00<00:00, 165.43it/s]
Validation DataLoader 0: 49%|████▊ | 152/313 [00:00<00:00, 165.44it/s]
Validation DataLoader 0: 49%|████▉ | 153/313 [00:00<00:00, 165.44it/s]
Validation DataLoader 0: 49%|████▉ | 154/313 [00:00<00:00, 165.42it/s]
Validation DataLoader 0: 50%|████▉ | 155/313 [00:00<00:00, 165.44it/s]
Validation DataLoader 0: 50%|████▉ | 156/313 [00:00<00:00, 165.42it/s]
Validation DataLoader 0: 50%|█████ | 157/313 [00:00<00:00, 165.42it/s]
Validation DataLoader 0: 50%|█████ | 158/313 [00:00<00:00, 165.41it/s]
Validation DataLoader 0: 51%|█████ | 159/313 [00:00<00:00, 165.38it/s]
Validation DataLoader 0: 51%|█████ | 160/313 [00:00<00:00, 165.39it/s]
Validation DataLoader 0: 51%|█████▏ | 161/313 [00:00<00:00, 165.39it/s]
Validation DataLoader 0: 52%|█████▏ | 162/313 [00:00<00:00, 165.38it/s]
Validation DataLoader 0: 52%|█████▏ | 163/313 [00:00<00:00, 165.38it/s]
Validation DataLoader 0: 52%|█████▏ | 164/313 [00:00<00:00, 165.39it/s]
Validation DataLoader 0: 53%|█████▎ | 165/313 [00:00<00:00, 165.39it/s]
Validation DataLoader 0: 53%|█████▎ | 166/313 [00:01<00:00, 165.35it/s]
Validation DataLoader 0: 53%|█████▎ | 167/313 [00:01<00:00, 165.36it/s]
Validation DataLoader 0: 54%|█████▎ | 168/313 [00:01<00:00, 165.38it/s]
Validation DataLoader 0: 54%|█████▍ | 169/313 [00:01<00:00, 165.40it/s]
Validation DataLoader 0: 54%|█████▍ | 170/313 [00:01<00:00, 165.41it/s]
Validation DataLoader 0: 55%|█████▍ | 171/313 [00:01<00:00, 165.41it/s]
Validation DataLoader 0: 55%|█████▍ | 172/313 [00:01<00:00, 165.43it/s]
Validation DataLoader 0: 55%|█████▌ | 173/313 [00:01<00:00, 165.44it/s]
Validation DataLoader 0: 56%|█████▌ | 174/313 [00:01<00:00, 165.45it/s]
Validation DataLoader 0: 56%|█████▌ | 175/313 [00:01<00:00, 165.45it/s]
Validation DataLoader 0: 56%|█████▌ | 176/313 [00:01<00:00, 165.45it/s]
Validation DataLoader 0: 57%|█████▋ | 177/313 [00:01<00:00, 165.46it/s]
Validation DataLoader 0: 57%|█████▋ | 178/313 [00:01<00:00, 165.46it/s]
Validation DataLoader 0: 57%|█████▋ | 179/313 [00:01<00:00, 165.46it/s]
Validation DataLoader 0: 58%|█████▊ | 180/313 [00:01<00:00, 165.46it/s]
Validation DataLoader 0: 58%|█████▊ | 181/313 [00:01<00:00, 165.46it/s]
Validation DataLoader 0: 58%|█████▊ | 182/313 [00:01<00:00, 165.48it/s]
Validation DataLoader 0: 58%|█████▊ | 183/313 [00:01<00:00, 165.48it/s]
Validation DataLoader 0: 59%|█████▉ | 184/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 59%|█████▉ | 185/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 59%|█████▉ | 186/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 60%|█████▉ | 187/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 60%|██████ | 188/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 60%|██████ | 189/313 [00:01<00:00, 165.50it/s]
Validation DataLoader 0: 61%|██████ | 190/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 61%|██████ | 191/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 61%|██████▏ | 192/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 62%|██████▏ | 193/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 62%|██████▏ | 194/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 62%|██████▏ | 195/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 63%|██████▎ | 196/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 63%|██████▎ | 197/313 [00:01<00:00, 165.48it/s]
Validation DataLoader 0: 63%|██████▎ | 198/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 64%|██████▎ | 199/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 64%|██████▍ | 200/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 64%|██████▍ | 201/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 65%|██████▍ | 202/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 65%|██████▍ | 203/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 65%|██████▌ | 204/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 65%|██████▌ | 205/313 [00:01<00:00, 165.50it/s]
Validation DataLoader 0: 66%|██████▌ | 206/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 66%|██████▌ | 207/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 66%|██████▋ | 208/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 67%|██████▋ | 209/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 67%|██████▋ | 210/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 67%|██████▋ | 211/313 [00:01<00:00, 165.50it/s]
Validation DataLoader 0: 68%|██████▊ | 212/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 68%|██████▊ | 213/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 68%|██████▊ | 214/313 [00:01<00:00, 165.50it/s]
Validation DataLoader 0: 69%|██████▊ | 215/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 69%|██████▉ | 216/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 69%|██████▉ | 217/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 70%|██████▉ | 218/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 70%|██████▉ | 219/313 [00:01<00:00, 165.55it/s]
Validation DataLoader 0: 70%|███████ | 220/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 71%|███████ | 221/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 71%|███████ | 222/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 71%|███████ | 223/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 72%|███████▏ | 224/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 72%|███████▏ | 225/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 72%|███████▏ | 226/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 73%|███████▎ | 227/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 73%|███████▎ | 228/313 [00:01<00:00, 165.58it/s]
Validation DataLoader 0: 73%|███████▎ | 229/313 [00:01<00:00, 165.58it/s]
Validation DataLoader 0: 73%|███████▎ | 230/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 74%|███████▍ | 231/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 74%|███████▍ | 232/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 74%|███████▍ | 233/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 75%|███████▍ | 234/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 75%|███████▌ | 235/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 75%|███████▌ | 236/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 76%|███████▌ | 237/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 76%|███████▌ | 238/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 76%|███████▋ | 239/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 77%|███████▋ | 240/313 [00:01<00:00, 165.55it/s]
Validation DataLoader 0: 77%|███████▋ | 241/313 [00:01<00:00, 165.55it/s]
Validation DataLoader 0: 77%|███████▋ | 242/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 78%|███████▊ | 243/313 [00:01<00:00, 165.55it/s]
Validation DataLoader 0: 78%|███████▊ | 244/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 78%|███████▊ | 245/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 79%|███████▊ | 246/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 79%|███████▉ | 247/313 [00:01<00:00, 165.49it/s]
Validation DataLoader 0: 79%|███████▉ | 248/313 [00:01<00:00, 165.50it/s]
Validation DataLoader 0: 80%|███████▉ | 249/313 [00:01<00:00, 165.51it/s]
Validation DataLoader 0: 80%|███████▉ | 250/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 80%|████████ | 251/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 81%|████████ | 252/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 81%|████████ | 253/313 [00:01<00:00, 165.52it/s]
Validation DataLoader 0: 81%|████████ | 254/313 [00:01<00:00, 165.53it/s]
Validation DataLoader 0: 81%|████████▏ | 255/313 [00:01<00:00, 165.53it/s]
Validation DataLoader 0: 82%|████████▏ | 256/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 82%|████████▏ | 257/313 [00:01<00:00, 165.54it/s]
Validation DataLoader 0: 82%|████████▏ | 258/313 [00:01<00:00, 165.55it/s]
Validation DataLoader 0: 83%|████████▎ | 259/313 [00:01<00:00, 165.56it/s]
Validation DataLoader 0: 83%|████████▎ | 260/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 83%|████████▎ | 261/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 84%|████████▎ | 262/313 [00:01<00:00, 165.58it/s]
Validation DataLoader 0: 84%|████████▍ | 263/313 [00:01<00:00, 165.59it/s]
Validation DataLoader 0: 84%|████████▍ | 264/313 [00:01<00:00, 165.59it/s]
Validation DataLoader 0: 85%|████████▍ | 265/313 [00:01<00:00, 165.59it/s]
Validation DataLoader 0: 85%|████████▍ | 266/313 [00:01<00:00, 165.59it/s]
Validation DataLoader 0: 85%|████████▌ | 267/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 86%|████████▌ | 268/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 86%|████████▌ | 269/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 86%|████████▋ | 270/313 [00:01<00:00, 165.57it/s]
Validation DataLoader 0: 87%|████████▋ | 271/313 [00:01<00:00, 165.43it/s]
Validation DataLoader 0: 87%|████████▋ | 272/313 [00:01<00:00, 165.38it/s]
Validation DataLoader 0: 87%|████████▋ | 273/313 [00:01<00:00, 165.30it/s]
Validation DataLoader 0: 88%|████████▊ | 274/313 [00:01<00:00, 165.27it/s]
Validation DataLoader 0: 88%|████████▊ | 275/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 88%|████████▊ | 276/313 [00:01<00:00, 165.12it/s]
Validation DataLoader 0: 88%|████████▊ | 277/313 [00:01<00:00, 165.03it/s]
Validation DataLoader 0: 89%|████████▉ | 278/313 [00:01<00:00, 165.01it/s]
Validation DataLoader 0: 89%|████████▉ | 279/313 [00:01<00:00, 164.93it/s]
Validation DataLoader 0: 89%|████████▉ | 280/313 [00:01<00:00, 164.90it/s]
Validation DataLoader 0: 90%|████████▉ | 281/313 [00:01<00:00, 164.92it/s]
Validation DataLoader 0: 90%|█████████ | 282/313 [00:01<00:00, 164.92it/s]
Validation DataLoader 0: 90%|█████████ | 283/313 [00:01<00:00, 164.90it/s]
Validation DataLoader 0: 91%|█████████ | 284/313 [00:01<00:00, 164.88it/s]
Validation DataLoader 0: 91%|█████████ | 285/313 [00:01<00:00, 164.89it/s]
Validation DataLoader 0: 91%|█████████▏| 286/313 [00:01<00:00, 164.90it/s]
Validation DataLoader 0: 92%|█████████▏| 287/313 [00:01<00:00, 164.90it/s]
Validation DataLoader 0: 92%|█████████▏| 288/313 [00:01<00:00, 164.91it/s]
Validation DataLoader 0: 92%|█████████▏| 289/313 [00:01<00:00, 164.91it/s]
Validation DataLoader 0: 93%|█████████▎| 290/313 [00:01<00:00, 164.92it/s]
Validation DataLoader 0: 93%|█████████▎| 291/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 93%|█████████▎| 292/313 [00:01<00:00, 164.95it/s]
Validation DataLoader 0: 94%|█████████▎| 293/313 [00:01<00:00, 164.97it/s]
Validation DataLoader 0: 94%|█████████▍| 294/313 [00:01<00:00, 164.97it/s]
Validation DataLoader 0: 94%|█████████▍| 295/313 [00:01<00:00, 164.96it/s]
Validation DataLoader 0: 95%|█████████▍| 296/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 95%|█████████▍| 297/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 95%|█████████▌| 298/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 96%|█████████▌| 299/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 96%|█████████▌| 300/313 [00:01<00:00, 164.94it/s]
Validation DataLoader 0: 96%|█████████▌| 301/313 [00:01<00:00, 164.86it/s]
Validation DataLoader 0: 96%|█████████▋| 302/313 [00:01<00:00, 164.80it/s]
Validation DataLoader 0: 97%|█████████▋| 303/313 [00:01<00:00, 164.73it/s]
Validation DataLoader 0: 97%|█████████▋| 304/313 [00:01<00:00, 164.67it/s]
Validation DataLoader 0: 97%|█████████▋| 305/313 [00:01<00:00, 164.61it/s]
Validation DataLoader 0: 98%|█████████▊| 306/313 [00:01<00:00, 164.58it/s]
Validation DataLoader 0: 98%|█████████▊| 307/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 98%|█████████▊| 308/313 [00:01<00:00, 164.48it/s]
Validation DataLoader 0: 99%|█████████▊| 309/313 [00:01<00:00, 164.44it/s]
Validation DataLoader 0: 99%|█████████▉| 310/313 [00:01<00:00, 164.38it/s]
Validation DataLoader 0: 99%|█████████▉| 311/313 [00:01<00:00, 164.33it/s]
Validation DataLoader 0: 100%|█████████▉| 312/313 [00:01<00:00, 164.33it/s]
Validation DataLoader 0: 100%|██████████| 313/313 [00:01<00:00, 164.47it/s]
Epoch 1: 100%|██████████| 3/3 [00:01<00:00, 1.55it/s, v_num=fp9x]
→ creating new artifact version for key 'testmodels/wandb/1c84fp9x.ckpt' in storage '/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops'
Epoch 1: 100%|██████████| 3/3 [00:01<00:00, 1.52it/s, v_num=fp9x]
Epoch 1: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 2: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 2: 33%|███▎ | 1/3 [00:00<00:00, 127.31it/s, v_num=fp9x]
Epoch 2: 33%|███▎ | 1/3 [00:00<00:00, 115.50it/s, v_num=fp9x]
Epoch 2: 67%|██████▋ | 2/3 [00:00<00:00, 131.78it/s, v_num=fp9x]
Epoch 2: 67%|██████▋ | 2/3 [00:00<00:00, 125.42it/s, v_num=fp9x]
Epoch 2: 100%|██████████| 3/3 [00:00<00:00, 134.56it/s, v_num=fp9x]
Epoch 2: 100%|██████████| 3/3 [00:00<00:00, 130.23it/s, v_num=fp9x]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: | | 0/? [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/313 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 1/313 [00:00<00:01, 199.49it/s]
Validation DataLoader 0: 1%| | 2/313 [00:00<00:01, 174.60it/s]
Validation DataLoader 0: 1%| | 3/313 [00:00<00:01, 164.76it/s]
Validation DataLoader 0: 1%|▏ | 4/313 [00:00<00:01, 160.62it/s]
Validation DataLoader 0: 2%|▏ | 5/313 [00:00<00:01, 159.63it/s]
Validation DataLoader 0: 2%|▏ | 6/313 [00:00<00:01, 160.50it/s]
Validation DataLoader 0: 2%|▏ | 7/313 [00:00<00:01, 161.13it/s]
Validation DataLoader 0: 3%|▎ | 8/313 [00:00<00:01, 161.40it/s]
Validation DataLoader 0: 3%|▎ | 9/313 [00:00<00:01, 161.83it/s]
Validation DataLoader 0: 3%|▎ | 10/313 [00:00<00:01, 162.22it/s]
Validation DataLoader 0: 4%|▎ | 11/313 [00:00<00:01, 162.58it/s]
Validation DataLoader 0: 4%|▍ | 12/313 [00:00<00:01, 163.13it/s]
Validation DataLoader 0: 4%|▍ | 13/313 [00:00<00:01, 163.57it/s]
Validation DataLoader 0: 4%|▍ | 14/313 [00:00<00:01, 163.84it/s]
Validation DataLoader 0: 5%|▍ | 15/313 [00:00<00:01, 164.02it/s]
Validation DataLoader 0: 5%|▌ | 16/313 [00:00<00:01, 164.09it/s]
Validation DataLoader 0: 5%|▌ | 17/313 [00:00<00:01, 164.24it/s]
Validation DataLoader 0: 6%|▌ | 18/313 [00:00<00:01, 164.10it/s]
Validation DataLoader 0: 6%|▌ | 19/313 [00:00<00:01, 164.05it/s]
Validation DataLoader 0: 6%|▋ | 20/313 [00:00<00:01, 164.02it/s]
Validation DataLoader 0: 7%|▋ | 21/313 [00:00<00:01, 163.92it/s]
Validation DataLoader 0: 7%|▋ | 22/313 [00:00<00:01, 164.07it/s]
Validation DataLoader 0: 7%|▋ | 23/313 [00:00<00:01, 164.06it/s]
Validation DataLoader 0: 8%|▊ | 24/313 [00:00<00:01, 164.30it/s]
Validation DataLoader 0: 8%|▊ | 25/313 [00:00<00:01, 164.42it/s]
Validation DataLoader 0: 8%|▊ | 26/313 [00:00<00:01, 164.59it/s]
Validation DataLoader 0: 9%|▊ | 27/313 [00:00<00:01, 164.72it/s]
Validation DataLoader 0: 9%|▉ | 28/313 [00:00<00:01, 164.81it/s]
Validation DataLoader 0: 9%|▉ | 29/313 [00:00<00:01, 164.90it/s]
Validation DataLoader 0: 10%|▉ | 30/313 [00:00<00:01, 164.82it/s]
Validation DataLoader 0: 10%|▉ | 31/313 [00:00<00:01, 164.94it/s]
Validation DataLoader 0: 10%|█ | 32/313 [00:00<00:01, 164.94it/s]
Validation DataLoader 0: 11%|█ | 33/313 [00:00<00:01, 165.00it/s]
Validation DataLoader 0: 11%|█ | 34/313 [00:00<00:01, 164.75it/s]
Validation DataLoader 0: 11%|█ | 35/313 [00:00<00:01, 164.69it/s]
Validation DataLoader 0: 12%|█▏ | 36/313 [00:00<00:01, 164.75it/s]
Validation DataLoader 0: 12%|█▏ | 37/313 [00:00<00:01, 164.81it/s]
Validation DataLoader 0: 12%|█▏ | 38/313 [00:00<00:01, 164.93it/s]
Validation DataLoader 0: 12%|█▏ | 39/313 [00:00<00:01, 164.90it/s]
Validation DataLoader 0: 13%|█▎ | 40/313 [00:00<00:01, 164.96it/s]
Validation DataLoader 0: 13%|█▎ | 41/313 [00:00<00:01, 164.94it/s]
Validation DataLoader 0: 13%|█▎ | 42/313 [00:00<00:01, 164.99it/s]
Validation DataLoader 0: 14%|█▎ | 43/313 [00:00<00:01, 165.08it/s]
Validation DataLoader 0: 14%|█▍ | 44/313 [00:00<00:01, 165.06it/s]
Validation DataLoader 0: 14%|█▍ | 45/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 15%|█▍ | 46/313 [00:00<00:01, 165.05it/s]
Validation DataLoader 0: 15%|█▌ | 47/313 [00:00<00:01, 165.16it/s]
Validation DataLoader 0: 15%|█▌ | 48/313 [00:00<00:01, 165.11it/s]
Validation DataLoader 0: 16%|█▌ | 49/313 [00:00<00:01, 165.20it/s]
Validation DataLoader 0: 16%|█▌ | 50/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 16%|█▋ | 51/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 17%|█▋ | 52/313 [00:00<00:01, 165.14it/s]
Validation DataLoader 0: 17%|█▋ | 53/313 [00:00<00:01, 165.17it/s]
Validation DataLoader 0: 17%|█▋ | 54/313 [00:00<00:01, 165.17it/s]
Validation DataLoader 0: 18%|█▊ | 55/313 [00:00<00:01, 165.15it/s]
Validation DataLoader 0: 18%|█▊ | 56/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 18%|█▊ | 57/313 [00:00<00:01, 165.18it/s]
Validation DataLoader 0: 19%|█▊ | 58/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 19%|█▉ | 59/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 19%|█▉ | 60/313 [00:00<00:01, 165.25it/s]
Validation DataLoader 0: 19%|█▉ | 61/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 20%|█▉ | 62/313 [00:00<00:01, 165.29it/s]
Validation DataLoader 0: 20%|██ | 63/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 20%|██ | 64/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 21%|██ | 65/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 21%|██ | 66/313 [00:00<00:01, 165.37it/s]
Validation DataLoader 0: 21%|██▏ | 67/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 22%|██▏ | 68/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 22%|██▏ | 69/313 [00:00<00:01, 165.34it/s]
Validation DataLoader 0: 22%|██▏ | 70/313 [00:00<00:01, 165.40it/s]
Validation DataLoader 0: 23%|██▎ | 71/313 [00:00<00:01, 165.17it/s]
Validation DataLoader 0: 23%|██▎ | 72/313 [00:00<00:01, 165.16it/s]
Validation DataLoader 0: 23%|██▎ | 73/313 [00:00<00:01, 165.18it/s]
Validation DataLoader 0: 24%|██▎ | 74/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 24%|██▍ | 75/313 [00:00<00:01, 165.25it/s]
Validation DataLoader 0: 24%|██▍ | 76/313 [00:00<00:01, 165.28it/s]
Validation DataLoader 0: 25%|██▍ | 77/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 25%|██▍ | 78/313 [00:00<00:01, 165.25it/s]
Validation DataLoader 0: 25%|██▌ | 79/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 26%|██▌ | 80/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 26%|██▌ | 81/313 [00:00<00:01, 165.26it/s]
Validation DataLoader 0: 26%|██▌ | 82/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 27%|██▋ | 83/313 [00:00<00:01, 165.36it/s]
Validation DataLoader 0: 27%|██▋ | 84/313 [00:00<00:01, 165.38it/s]
Validation DataLoader 0: 27%|██▋ | 85/313 [00:00<00:01, 165.09it/s]
Validation DataLoader 0: 27%|██▋ | 86/313 [00:00<00:01, 165.13it/s]
Validation DataLoader 0: 28%|██▊ | 87/313 [00:00<00:01, 165.15it/s]
Validation DataLoader 0: 28%|██▊ | 88/313 [00:00<00:01, 165.18it/s]
Validation DataLoader 0: 28%|██▊ | 89/313 [00:00<00:01, 165.19it/s]
Validation DataLoader 0: 29%|██▉ | 90/313 [00:00<00:01, 165.17it/s]
Validation DataLoader 0: 29%|██▉ | 91/313 [00:00<00:01, 165.19it/s]
Validation DataLoader 0: 29%|██▉ | 92/313 [00:00<00:01, 165.22it/s]
Validation DataLoader 0: 30%|██▉ | 93/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 30%|███ | 94/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 30%|███ | 95/313 [00:00<00:01, 165.17it/s]
Validation DataLoader 0: 31%|███ | 96/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 31%|███ | 97/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 31%|███▏ | 98/313 [00:00<00:01, 165.27it/s]
Validation DataLoader 0: 32%|███▏ | 99/313 [00:00<00:01, 165.28it/s]
Validation DataLoader 0: 32%|███▏ | 100/313 [00:00<00:01, 165.25it/s]
Validation DataLoader 0: 32%|███▏ | 101/313 [00:00<00:01, 165.30it/s]
Validation DataLoader 0: 33%|███▎ | 102/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 33%|███▎ | 103/313 [00:00<00:01, 165.24it/s]
Validation DataLoader 0: 33%|███▎ | 104/313 [00:00<00:01, 165.27it/s]
Validation DataLoader 0: 34%|███▎ | 105/313 [00:00<00:01, 165.15it/s]
Validation DataLoader 0: 34%|███▍ | 106/313 [00:00<00:01, 165.18it/s]
Validation DataLoader 0: 34%|███▍ | 107/313 [00:00<00:01, 165.21it/s]
Validation DataLoader 0: 35%|███▍ | 108/313 [00:00<00:01, 165.20it/s]
Validation DataLoader 0: 35%|███▍ | 109/313 [00:00<00:01, 164.87it/s]
Validation DataLoader 0: 35%|███▌ | 110/313 [00:00<00:01, 164.87it/s]
Validation DataLoader 0: 35%|███▌ | 111/313 [00:00<00:01, 164.88it/s]
Validation DataLoader 0: 36%|███▌ | 112/313 [00:00<00:01, 164.91it/s]
Validation DataLoader 0: 36%|███▌ | 113/313 [00:00<00:01, 164.87it/s]
Validation DataLoader 0: 36%|███▋ | 114/313 [00:00<00:01, 164.90it/s]
Validation DataLoader 0: 37%|███▋ | 115/313 [00:00<00:01, 164.86it/s]
Validation DataLoader 0: 37%|███▋ | 116/313 [00:00<00:01, 164.86it/s]
Validation DataLoader 0: 37%|███▋ | 117/313 [00:00<00:01, 164.85it/s]
Validation DataLoader 0: 38%|███▊ | 118/313 [00:00<00:01, 164.84it/s]
Validation DataLoader 0: 38%|███▊ | 119/313 [00:00<00:01, 164.86it/s]
Validation DataLoader 0: 38%|███▊ | 120/313 [00:00<00:01, 164.88it/s]
Validation DataLoader 0: 39%|███▊ | 121/313 [00:00<00:01, 164.84it/s]
Validation DataLoader 0: 39%|███▉ | 122/313 [00:00<00:01, 164.82it/s]
Validation DataLoader 0: 39%|███▉ | 123/313 [00:00<00:01, 164.84it/s]
Validation DataLoader 0: 40%|███▉ | 124/313 [00:00<00:01, 164.89it/s]
Validation DataLoader 0: 40%|███▉ | 125/313 [00:00<00:01, 164.86it/s]
Validation DataLoader 0: 40%|████ | 126/313 [00:00<00:01, 164.93it/s]
Validation DataLoader 0: 41%|████ | 127/313 [00:00<00:01, 164.91it/s]
Validation DataLoader 0: 41%|████ | 128/313 [00:00<00:01, 164.94it/s]
Validation DataLoader 0: 41%|████ | 129/313 [00:00<00:01, 164.97it/s]
Validation DataLoader 0: 42%|████▏ | 130/313 [00:00<00:01, 164.99it/s]
Validation DataLoader 0: 42%|████▏ | 131/313 [00:00<00:01, 165.01it/s]
Validation DataLoader 0: 42%|████▏ | 132/313 [00:00<00:01, 165.01it/s]
Validation DataLoader 0: 42%|████▏ | 133/313 [00:00<00:01, 165.03it/s]
Validation DataLoader 0: 43%|████▎ | 134/313 [00:00<00:01, 165.04it/s]
Validation DataLoader 0: 43%|████▎ | 135/313 [00:00<00:01, 165.03it/s]
Validation DataLoader 0: 43%|████▎ | 136/313 [00:00<00:01, 165.01it/s]
Validation DataLoader 0: 44%|████▍ | 137/313 [00:00<00:01, 165.06it/s]
Validation DataLoader 0: 44%|████▍ | 138/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 44%|████▍ | 139/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 45%|████▍ | 140/313 [00:00<00:01, 165.08it/s]
Validation DataLoader 0: 45%|████▌ | 141/313 [00:00<00:01, 165.09it/s]
Validation DataLoader 0: 45%|████▌ | 142/313 [00:00<00:01, 165.13it/s]
Validation DataLoader 0: 46%|████▌ | 143/313 [00:00<00:01, 165.11it/s]
Validation DataLoader 0: 46%|████▌ | 144/313 [00:00<00:01, 165.12it/s]
Validation DataLoader 0: 46%|████▋ | 145/313 [00:00<00:01, 165.08it/s]
Validation DataLoader 0: 47%|████▋ | 146/313 [00:00<00:01, 165.08it/s]
Validation DataLoader 0: 47%|████▋ | 147/313 [00:00<00:01, 165.08it/s]
Validation DataLoader 0: 47%|████▋ | 148/313 [00:00<00:00, 165.07it/s]
Validation DataLoader 0: 48%|████▊ | 149/313 [00:00<00:00, 164.98it/s]
Validation DataLoader 0: 48%|████▊ | 150/313 [00:00<00:00, 165.02it/s]
Validation DataLoader 0: 48%|████▊ | 151/313 [00:00<00:00, 165.02it/s]
Validation DataLoader 0: 49%|████▊ | 152/313 [00:00<00:00, 165.03it/s]
Validation DataLoader 0: 49%|████▉ | 153/313 [00:00<00:00, 165.02it/s]
Validation DataLoader 0: 49%|████▉ | 154/313 [00:00<00:00, 165.04it/s]
Validation DataLoader 0: 50%|████▉ | 155/313 [00:00<00:00, 165.05it/s]
Validation DataLoader 0: 50%|████▉ | 156/313 [00:00<00:00, 165.05it/s]
Validation DataLoader 0: 50%|█████ | 157/313 [00:00<00:00, 165.05it/s]
Validation DataLoader 0: 50%|█████ | 158/313 [00:00<00:00, 165.06it/s]
Validation DataLoader 0: 51%|█████ | 159/313 [00:00<00:00, 165.06it/s]
Validation DataLoader 0: 51%|█████ | 160/313 [00:00<00:00, 165.06it/s]
Validation DataLoader 0: 51%|█████▏ | 161/313 [00:00<00:00, 165.03it/s]
Validation DataLoader 0: 52%|█████▏ | 162/313 [00:00<00:00, 165.07it/s]
Validation DataLoader 0: 52%|█████▏ | 163/313 [00:00<00:00, 165.08it/s]
Validation DataLoader 0: 52%|█████▏ | 164/313 [00:00<00:00, 165.10it/s]
Validation DataLoader 0: 53%|█████▎ | 165/313 [00:00<00:00, 165.11it/s]
Validation DataLoader 0: 53%|█████▎ | 166/313 [00:01<00:00, 165.11it/s]
Validation DataLoader 0: 53%|█████▎ | 167/313 [00:01<00:00, 165.11it/s]
Validation DataLoader 0: 54%|█████▎ | 168/313 [00:01<00:00, 165.12it/s]
Validation DataLoader 0: 54%|█████▍ | 169/313 [00:01<00:00, 165.13it/s]
Validation DataLoader 0: 54%|█████▍ | 170/313 [00:01<00:00, 165.14it/s]
Validation DataLoader 0: 55%|█████▍ | 171/313 [00:01<00:00, 165.15it/s]
Validation DataLoader 0: 55%|█████▍ | 172/313 [00:01<00:00, 165.15it/s]
Validation DataLoader 0: 55%|█████▌ | 173/313 [00:01<00:00, 165.16it/s]
Validation DataLoader 0: 56%|█████▌ | 174/313 [00:01<00:00, 165.17it/s]
Validation DataLoader 0: 56%|█████▌ | 175/313 [00:01<00:00, 165.18it/s]
Validation DataLoader 0: 56%|█████▌ | 176/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 57%|█████▋ | 177/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 57%|█████▋ | 178/313 [00:01<00:00, 165.22it/s]
Validation DataLoader 0: 57%|█████▋ | 179/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 58%|█████▊ | 180/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 58%|█████▊ | 181/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 58%|█████▊ | 182/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 58%|█████▊ | 183/313 [00:01<00:00, 165.27it/s]
Validation DataLoader 0: 59%|█████▉ | 184/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 59%|█████▉ | 185/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 59%|█████▉ | 186/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 60%|█████▉ | 187/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 60%|██████ | 188/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 60%|██████ | 189/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 61%|██████ | 190/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 61%|██████ | 191/313 [00:01<00:00, 165.24it/s]
Validation DataLoader 0: 61%|██████▏ | 192/313 [00:01<00:00, 165.23it/s]
Validation DataLoader 0: 62%|██████▏ | 193/313 [00:01<00:00, 165.23it/s]
Validation DataLoader 0: 62%|██████▏ | 194/313 [00:01<00:00, 165.22it/s]
Validation DataLoader 0: 62%|██████▏ | 195/313 [00:01<00:00, 165.21it/s]
Validation DataLoader 0: 63%|██████▎ | 196/313 [00:01<00:00, 165.21it/s]
Validation DataLoader 0: 63%|██████▎ | 197/313 [00:01<00:00, 165.22it/s]
Validation DataLoader 0: 63%|██████▎ | 198/313 [00:01<00:00, 165.16it/s]
Validation DataLoader 0: 64%|██████▎ | 199/313 [00:01<00:00, 165.14it/s]
Validation DataLoader 0: 64%|██████▍ | 200/313 [00:01<00:00, 165.14it/s]
Validation DataLoader 0: 64%|██████▍ | 201/313 [00:01<00:00, 165.15it/s]
Validation DataLoader 0: 65%|██████▍ | 202/313 [00:01<00:00, 165.17it/s]
Validation DataLoader 0: 65%|██████▍ | 203/313 [00:01<00:00, 165.18it/s]
Validation DataLoader 0: 65%|██████▌ | 204/313 [00:01<00:00, 165.19it/s]
Validation DataLoader 0: 65%|██████▌ | 205/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 66%|██████▌ | 206/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 66%|██████▌ | 207/313 [00:01<00:00, 165.21it/s]
Validation DataLoader 0: 66%|██████▋ | 208/313 [00:01<00:00, 165.21it/s]
Validation DataLoader 0: 67%|██████▋ | 209/313 [00:01<00:00, 165.21it/s]
Validation DataLoader 0: 67%|██████▋ | 210/313 [00:01<00:00, 165.22it/s]
Validation DataLoader 0: 67%|██████▋ | 211/313 [00:01<00:00, 165.20it/s]
Validation DataLoader 0: 68%|██████▊ | 212/313 [00:01<00:00, 165.25it/s]
Validation DataLoader 0: 68%|██████▊ | 213/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 68%|██████▊ | 214/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 69%|██████▊ | 215/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 69%|██████▉ | 216/313 [00:01<00:00, 165.27it/s]
Validation DataLoader 0: 69%|██████▉ | 217/313 [00:01<00:00, 165.27it/s]
Validation DataLoader 0: 70%|██████▉ | 218/313 [00:01<00:00, 165.26it/s]
Validation DataLoader 0: 70%|██████▉ | 219/313 [00:01<00:00, 165.28it/s]
Validation DataLoader 0: 70%|███████ | 220/313 [00:01<00:00, 165.29it/s]
Validation DataLoader 0: 71%|███████ | 221/313 [00:01<00:00, 165.29it/s]
Validation DataLoader 0: 71%|███████ | 222/313 [00:01<00:00, 165.30it/s]
Validation DataLoader 0: 71%|███████ | 223/313 [00:01<00:00, 165.22it/s]
Validation DataLoader 0: 72%|███████▏ | 224/313 [00:01<00:00, 164.89it/s]
Validation DataLoader 0: 72%|███████▏ | 225/313 [00:01<00:00, 164.71it/s]
Validation DataLoader 0: 72%|███████▏ | 226/313 [00:01<00:00, 164.65it/s]
Validation DataLoader 0: 73%|███████▎ | 227/313 [00:01<00:00, 164.56it/s]
Validation DataLoader 0: 73%|███████▎ | 228/313 [00:01<00:00, 164.52it/s]
Validation DataLoader 0: 73%|███████▎ | 229/313 [00:01<00:00, 164.52it/s]
Validation DataLoader 0: 73%|███████▎ | 230/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 74%|███████▍ | 231/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 74%|███████▍ | 232/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 74%|███████▍ | 233/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 75%|███████▍ | 234/313 [00:01<00:00, 164.49it/s]
Validation DataLoader 0: 75%|███████▌ | 235/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 75%|███████▌ | 236/313 [00:01<00:00, 164.50it/s]
Validation DataLoader 0: 76%|███████▌ | 237/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 76%|███████▌ | 238/313 [00:01<00:00, 164.52it/s]
Validation DataLoader 0: 76%|███████▋ | 239/313 [00:01<00:00, 164.53it/s]
Validation DataLoader 0: 77%|███████▋ | 240/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 77%|███████▋ | 241/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 77%|███████▋ | 242/313 [00:01<00:00, 164.54it/s]
Validation DataLoader 0: 78%|███████▊ | 243/313 [00:01<00:00, 164.51it/s]
Validation DataLoader 0: 78%|███████▊ | 244/313 [00:01<00:00, 164.25it/s]
Validation DataLoader 0: 78%|███████▊ | 245/313 [00:01<00:00, 164.19it/s]
Validation DataLoader 0: 79%|███████▊ | 246/313 [00:01<00:00, 164.04it/s]
Validation DataLoader 0: 79%|███████▉ | 247/313 [00:01<00:00, 163.81it/s]
Validation DataLoader 0: 79%|███████▉ | 248/313 [00:01<00:00, 163.62it/s]
Validation DataLoader 0: 80%|███████▉ | 249/313 [00:01<00:00, 163.56it/s]
Validation DataLoader 0: 80%|███████▉ | 250/313 [00:01<00:00, 163.47it/s]
Validation DataLoader 0: 80%|████████ | 251/313 [00:01<00:00, 163.37it/s]
Validation DataLoader 0: 81%|████████ | 252/313 [00:01<00:00, 163.33it/s]
Validation DataLoader 0: 81%|████████ | 253/313 [00:01<00:00, 163.21it/s]
Validation DataLoader 0: 81%|████████ | 254/313 [00:01<00:00, 163.15it/s]
Validation DataLoader 0: 81%|████████▏ | 255/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 82%|████████▏ | 256/313 [00:01<00:00, 163.00it/s]
Validation DataLoader 0: 82%|████████▏ | 257/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 82%|████████▏ | 258/313 [00:01<00:00, 162.84it/s]
Validation DataLoader 0: 83%|████████▎ | 259/313 [00:01<00:00, 162.76it/s]
Validation DataLoader 0: 83%|████████▎ | 260/313 [00:01<00:00, 162.69it/s]
Validation DataLoader 0: 83%|████████▎ | 261/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 84%|████████▎ | 262/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 84%|████████▍ | 263/313 [00:01<00:00, 162.56it/s]
Validation DataLoader 0: 84%|████████▍ | 264/313 [00:01<00:00, 162.49it/s]
Validation DataLoader 0: 85%|████████▍ | 265/313 [00:01<00:00, 162.49it/s]
Validation DataLoader 0: 85%|████████▍ | 266/313 [00:01<00:00, 162.49it/s]
Validation DataLoader 0: 85%|████████▌ | 267/313 [00:01<00:00, 162.49it/s]
Validation DataLoader 0: 86%|████████▌ | 268/313 [00:01<00:00, 162.51it/s]
Validation DataLoader 0: 86%|████████▌ | 269/313 [00:01<00:00, 162.51it/s]
Validation DataLoader 0: 86%|████████▋ | 270/313 [00:01<00:00, 162.50it/s]
Validation DataLoader 0: 87%|████████▋ | 271/313 [00:01<00:00, 162.51it/s]
Validation DataLoader 0: 87%|████████▋ | 272/313 [00:01<00:00, 162.51it/s]
Validation DataLoader 0: 87%|████████▋ | 273/313 [00:01<00:00, 162.52it/s]
Validation DataLoader 0: 88%|████████▊ | 274/313 [00:01<00:00, 162.54it/s]
Validation DataLoader 0: 88%|████████▊ | 275/313 [00:01<00:00, 162.55it/s]
Validation DataLoader 0: 88%|████████▊ | 276/313 [00:01<00:00, 162.56it/s]
Validation DataLoader 0: 88%|████████▊ | 277/313 [00:01<00:00, 162.57it/s]
Validation DataLoader 0: 89%|████████▉ | 278/313 [00:01<00:00, 162.57it/s]
Validation DataLoader 0: 89%|████████▉ | 279/313 [00:01<00:00, 162.58it/s]
Validation DataLoader 0: 89%|████████▉ | 280/313 [00:01<00:00, 162.59it/s]
Validation DataLoader 0: 90%|████████▉ | 281/313 [00:01<00:00, 162.60it/s]
Validation DataLoader 0: 90%|█████████ | 282/313 [00:01<00:00, 162.59it/s]
Validation DataLoader 0: 90%|█████████ | 283/313 [00:01<00:00, 162.59it/s]
Validation DataLoader 0: 91%|█████████ | 284/313 [00:01<00:00, 162.60it/s]
Validation DataLoader 0: 91%|█████████ | 285/313 [00:01<00:00, 162.60it/s]
Validation DataLoader 0: 91%|█████████▏| 286/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 92%|█████████▏| 287/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 92%|█████████▏| 288/313 [00:01<00:00, 162.63it/s]
Validation DataLoader 0: 92%|█████████▏| 289/313 [00:01<00:00, 162.65it/s]
Validation DataLoader 0: 93%|█████████▎| 290/313 [00:01<00:00, 162.67it/s]
Validation DataLoader 0: 93%|█████████▎| 291/313 [00:01<00:00, 162.67it/s]
Validation DataLoader 0: 93%|█████████▎| 292/313 [00:01<00:00, 162.68it/s]
Validation DataLoader 0: 94%|█████████▎| 293/313 [00:01<00:00, 162.68it/s]
Validation DataLoader 0: 94%|█████████▍| 294/313 [00:01<00:00, 162.69it/s]
Validation DataLoader 0: 94%|█████████▍| 295/313 [00:01<00:00, 162.71it/s]
Validation DataLoader 0: 95%|█████████▍| 296/313 [00:01<00:00, 162.71it/s]
Validation DataLoader 0: 95%|█████████▍| 297/313 [00:01<00:00, 162.73it/s]
Validation DataLoader 0: 95%|█████████▌| 298/313 [00:01<00:00, 162.74it/s]
Validation DataLoader 0: 96%|█████████▌| 299/313 [00:01<00:00, 162.76it/s]
Validation DataLoader 0: 96%|█████████▌| 300/313 [00:01<00:00, 162.77it/s]
Validation DataLoader 0: 96%|█████████▌| 301/313 [00:01<00:00, 162.79it/s]
Validation DataLoader 0: 96%|█████████▋| 302/313 [00:01<00:00, 162.81it/s]
Validation DataLoader 0: 97%|█████████▋| 303/313 [00:01<00:00, 162.82it/s]
Validation DataLoader 0: 97%|█████████▋| 304/313 [00:01<00:00, 162.84it/s]
Validation DataLoader 0: 97%|█████████▋| 305/313 [00:01<00:00, 162.85it/s]
Validation DataLoader 0: 98%|█████████▊| 306/313 [00:01<00:00, 162.86it/s]
Validation DataLoader 0: 98%|█████████▊| 307/313 [00:01<00:00, 162.87it/s]
Validation DataLoader 0: 98%|█████████▊| 308/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 99%|█████████▊| 309/313 [00:01<00:00, 162.89it/s]
Validation DataLoader 0: 99%|█████████▉| 310/313 [00:01<00:00, 162.91it/s]
Validation DataLoader 0: 99%|█████████▉| 311/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 100%|█████████▉| 312/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 100%|██████████| 313/313 [00:01<00:00, 163.05it/s]
Epoch 2: 100%|██████████| 3/3 [00:01<00:00, 1.54it/s, v_num=fp9x]
→ creating new artifact version for key 'testmodels/wandb/1c84fp9x.ckpt' in storage '/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops'
Epoch 2: 100%|██████████| 3/3 [00:01<00:00, 1.51it/s, v_num=fp9x]
Epoch 2: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 3: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 3: 33%|███▎ | 1/3 [00:00<00:00, 121.77it/s, v_num=fp9x]
Epoch 3: 33%|███▎ | 1/3 [00:00<00:00, 114.69it/s, v_num=fp9x]
Epoch 3: 67%|██████▋ | 2/3 [00:00<00:00, 121.57it/s, v_num=fp9x]
Epoch 3: 67%|██████▋ | 2/3 [00:00<00:00, 117.92it/s, v_num=fp9x]
Epoch 3: 100%|██████████| 3/3 [00:00<00:00, 121.91it/s, v_num=fp9x]
Epoch 3: 100%|██████████| 3/3 [00:00<00:00, 118.12it/s, v_num=fp9x]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: | | 0/? [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/313 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 1/313 [00:00<00:01, 226.32it/s]
Validation DataLoader 0: 1%| | 2/313 [00:00<00:01, 182.46it/s]
Validation DataLoader 0: 1%| | 3/313 [00:00<00:01, 177.11it/s]
Validation DataLoader 0: 1%|▏ | 4/313 [00:00<00:01, 171.65it/s]
Validation DataLoader 0: 2%|▏ | 5/313 [00:00<00:01, 170.50it/s]
Validation DataLoader 0: 2%|▏ | 6/313 [00:00<00:01, 169.32it/s]
Validation DataLoader 0: 2%|▏ | 7/313 [00:00<00:01, 169.29it/s]
Validation DataLoader 0: 3%|▎ | 8/313 [00:00<00:01, 168.19it/s]
Validation DataLoader 0: 3%|▎ | 9/313 [00:00<00:01, 166.78it/s]
Validation DataLoader 0: 3%|▎ | 10/313 [00:00<00:01, 166.83it/s]
Validation DataLoader 0: 4%|▎ | 11/313 [00:00<00:01, 166.54it/s]
Validation DataLoader 0: 4%|▍ | 12/313 [00:00<00:01, 166.29it/s]
Validation DataLoader 0: 4%|▍ | 13/313 [00:00<00:01, 166.14it/s]
Validation DataLoader 0: 4%|▍ | 14/313 [00:00<00:01, 166.05it/s]
Validation DataLoader 0: 5%|▍ | 15/313 [00:00<00:01, 165.86it/s]
Validation DataLoader 0: 5%|▌ | 16/313 [00:00<00:01, 165.78it/s]
Validation DataLoader 0: 5%|▌ | 17/313 [00:00<00:01, 165.78it/s]
Validation DataLoader 0: 6%|▌ | 18/313 [00:00<00:01, 165.78it/s]
Validation DataLoader 0: 6%|▌ | 19/313 [00:00<00:01, 165.64it/s]
Validation DataLoader 0: 6%|▋ | 20/313 [00:00<00:01, 165.39it/s]
Validation DataLoader 0: 7%|▋ | 21/313 [00:00<00:01, 165.32it/s]
Validation DataLoader 0: 7%|▋ | 22/313 [00:00<00:01, 165.46it/s]
Validation DataLoader 0: 7%|▋ | 23/313 [00:00<00:01, 165.51it/s]
Validation DataLoader 0: 8%|▊ | 24/313 [00:00<00:01, 165.60it/s]
Validation DataLoader 0: 8%|▊ | 25/313 [00:00<00:01, 165.80it/s]
Validation DataLoader 0: 8%|▊ | 26/313 [00:00<00:01, 165.85it/s]
Validation DataLoader 0: 9%|▊ | 27/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 9%|▉ | 28/313 [00:00<00:01, 165.98it/s]
Validation DataLoader 0: 9%|▉ | 29/313 [00:00<00:01, 166.04it/s]
Validation DataLoader 0: 10%|▉ | 30/313 [00:00<00:01, 166.15it/s]
Validation DataLoader 0: 10%|▉ | 31/313 [00:00<00:01, 166.18it/s]
Validation DataLoader 0: 10%|█ | 32/313 [00:00<00:01, 166.28it/s]
Validation DataLoader 0: 11%|█ | 33/313 [00:00<00:01, 166.29it/s]
Validation DataLoader 0: 11%|█ | 34/313 [00:00<00:01, 166.19it/s]
Validation DataLoader 0: 11%|█ | 35/313 [00:00<00:01, 165.96it/s]
Validation DataLoader 0: 12%|█▏ | 36/313 [00:00<00:01, 165.85it/s]
Validation DataLoader 0: 12%|█▏ | 37/313 [00:00<00:01, 165.78it/s]
Validation DataLoader 0: 12%|█▏ | 38/313 [00:00<00:01, 165.83it/s]
Validation DataLoader 0: 12%|█▏ | 39/313 [00:00<00:01, 165.84it/s]
Validation DataLoader 0: 13%|█▎ | 40/313 [00:00<00:01, 165.86it/s]
Validation DataLoader 0: 13%|█▎ | 41/313 [00:00<00:01, 165.76it/s]
Validation DataLoader 0: 13%|█▎ | 42/313 [00:00<00:01, 165.75it/s]
Validation DataLoader 0: 14%|█▎ | 43/313 [00:00<00:01, 165.89it/s]
Validation DataLoader 0: 14%|█▍ | 44/313 [00:00<00:01, 165.85it/s]
Validation DataLoader 0: 14%|█▍ | 45/313 [00:00<00:01, 165.94it/s]
Validation DataLoader 0: 15%|█▍ | 46/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 15%|█▌ | 47/313 [00:00<00:01, 165.99it/s]
Validation DataLoader 0: 15%|█▌ | 48/313 [00:00<00:01, 166.00it/s]
Validation DataLoader 0: 16%|█▌ | 49/313 [00:00<00:01, 166.00it/s]
Validation DataLoader 0: 16%|█▌ | 50/313 [00:00<00:01, 165.96it/s]
Validation DataLoader 0: 16%|█▋ | 51/313 [00:00<00:01, 165.99it/s]
Validation DataLoader 0: 17%|█▋ | 52/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 17%|█▋ | 53/313 [00:00<00:01, 165.90it/s]
Validation DataLoader 0: 17%|█▋ | 54/313 [00:00<00:01, 165.90it/s]
Validation DataLoader 0: 18%|█▊ | 55/313 [00:00<00:01, 165.90it/s]
Validation DataLoader 0: 18%|█▊ | 56/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 18%|█▊ | 57/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 19%|█▊ | 58/313 [00:00<00:01, 165.94it/s]
Validation DataLoader 0: 19%|█▉ | 59/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 19%|█▉ | 60/313 [00:00<00:01, 165.97it/s]
Validation DataLoader 0: 19%|█▉ | 61/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 20%|█▉ | 62/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 20%|██ | 63/313 [00:00<00:01, 165.80it/s]
Validation DataLoader 0: 20%|██ | 64/313 [00:00<00:01, 165.92it/s]
Validation DataLoader 0: 21%|██ | 65/313 [00:00<00:01, 165.91it/s]
Validation DataLoader 0: 21%|██ | 66/313 [00:00<00:01, 165.94it/s]
Validation DataLoader 0: 21%|██▏ | 67/313 [00:00<00:01, 165.97it/s]
Validation DataLoader 0: 22%|██▏ | 68/313 [00:00<00:01, 165.95it/s]
Validation DataLoader 0: 22%|██▏ | 69/313 [00:00<00:01, 165.94it/s]
Validation DataLoader 0: 22%|██▏ | 70/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 23%|██▎ | 71/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 23%|██▎ | 72/313 [00:00<00:01, 165.93it/s]
Validation DataLoader 0: 23%|██▎ | 73/313 [00:00<00:01, 165.91it/s]
Validation DataLoader 0: 24%|██▎ | 74/313 [00:00<00:01, 165.77it/s]
Validation DataLoader 0: 24%|██▍ | 75/313 [00:00<00:01, 165.85it/s]
Validation DataLoader 0: 24%|██▍ | 76/313 [00:00<00:01, 165.84it/s]
Validation DataLoader 0: 25%|██▍ | 77/313 [00:00<00:01, 165.81it/s]
Validation DataLoader 0: 25%|██▍ | 78/313 [00:00<00:01, 165.82it/s]
Validation DataLoader 0: 25%|██▌ | 79/313 [00:00<00:01, 165.82it/s]
Validation DataLoader 0: 26%|██▌ | 80/313 [00:00<00:01, 165.81it/s]
Validation DataLoader 0: 26%|██▌ | 81/313 [00:00<00:01, 165.79it/s]
Validation DataLoader 0: 26%|██▌ | 82/313 [00:00<00:01, 165.83it/s]
Validation DataLoader 0: 27%|██▋ | 83/313 [00:00<00:01, 165.85it/s]
Validation DataLoader 0: 27%|██▋ | 84/313 [00:00<00:01, 165.84it/s]
Validation DataLoader 0: 27%|██▋ | 85/313 [00:00<00:01, 165.91it/s]
Validation DataLoader 0: 27%|██▋ | 86/313 [00:00<00:01, 165.72it/s]
Validation DataLoader 0: 28%|██▊ | 87/313 [00:00<00:01, 165.71it/s]
Validation DataLoader 0: 28%|██▊ | 88/313 [00:00<00:01, 165.50it/s]
Validation DataLoader 0: 28%|██▊ | 89/313 [00:00<00:01, 165.16it/s]
Validation DataLoader 0: 29%|██▉ | 90/313 [00:00<00:01, 164.89it/s]
Validation DataLoader 0: 29%|██▉ | 91/313 [00:00<00:01, 164.65it/s]
Validation DataLoader 0: 29%|██▉ | 92/313 [00:00<00:01, 164.37it/s]
Validation DataLoader 0: 30%|██▉ | 93/313 [00:00<00:01, 164.29it/s]
Validation DataLoader 0: 30%|███ | 94/313 [00:00<00:01, 164.09it/s]
Validation DataLoader 0: 30%|███ | 95/313 [00:00<00:01, 163.75it/s]
Validation DataLoader 0: 31%|███ | 96/313 [00:00<00:01, 163.76it/s]
Validation DataLoader 0: 31%|███ | 97/313 [00:00<00:01, 163.77it/s]
Validation DataLoader 0: 31%|███▏ | 98/313 [00:00<00:01, 163.77it/s]
Validation DataLoader 0: 32%|███▏ | 99/313 [00:00<00:01, 163.81it/s]
Validation DataLoader 0: 32%|███▏ | 100/313 [00:00<00:01, 163.94it/s]
Validation DataLoader 0: 32%|███▏ | 101/313 [00:00<00:01, 163.66it/s]
Validation DataLoader 0: 33%|███▎ | 102/313 [00:00<00:01, 163.49it/s]
Validation DataLoader 0: 33%|███▎ | 103/313 [00:00<00:01, 163.30it/s]
Validation DataLoader 0: 33%|███▎ | 104/313 [00:00<00:01, 163.25it/s]
Validation DataLoader 0: 34%|███▎ | 105/313 [00:00<00:01, 163.08it/s]
Validation DataLoader 0: 34%|███▍ | 106/313 [00:00<00:01, 162.88it/s]
Validation DataLoader 0: 34%|███▍ | 107/313 [00:00<00:01, 162.67it/s]
Validation DataLoader 0: 35%|███▍ | 108/313 [00:00<00:01, 162.46it/s]
Validation DataLoader 0: 35%|███▍ | 109/313 [00:00<00:01, 162.42it/s]
Validation DataLoader 0: 35%|███▌ | 110/313 [00:00<00:01, 162.23it/s]
Validation DataLoader 0: 35%|███▌ | 111/313 [00:00<00:01, 162.25it/s]
Validation DataLoader 0: 36%|███▌ | 112/313 [00:00<00:01, 162.26it/s]
Validation DataLoader 0: 36%|███▌ | 113/313 [00:00<00:01, 162.35it/s]
Validation DataLoader 0: 36%|███▋ | 114/313 [00:00<00:01, 162.19it/s]
Validation DataLoader 0: 37%|███▋ | 115/313 [00:00<00:01, 162.03it/s]
Validation DataLoader 0: 37%|███▋ | 116/313 [00:00<00:01, 161.94it/s]
Validation DataLoader 0: 37%|███▋ | 117/313 [00:00<00:01, 161.89it/s]
Validation DataLoader 0: 38%|███▊ | 118/313 [00:00<00:01, 161.88it/s]
Validation DataLoader 0: 38%|███▊ | 119/313 [00:00<00:01, 161.87it/s]
Validation DataLoader 0: 38%|███▊ | 120/313 [00:00<00:01, 161.87it/s]
Validation DataLoader 0: 39%|███▊ | 121/313 [00:00<00:01, 161.87it/s]
Validation DataLoader 0: 39%|███▉ | 122/313 [00:00<00:01, 161.92it/s]
Validation DataLoader 0: 39%|███▉ | 123/313 [00:00<00:01, 162.03it/s]
Validation DataLoader 0: 40%|███▉ | 124/313 [00:00<00:01, 162.09it/s]
Validation DataLoader 0: 40%|███▉ | 125/313 [00:00<00:01, 162.01it/s]
Validation DataLoader 0: 40%|████ | 126/313 [00:00<00:01, 162.03it/s]
Validation DataLoader 0: 41%|████ | 127/313 [00:00<00:01, 162.04it/s]
Validation DataLoader 0: 41%|████ | 128/313 [00:00<00:01, 162.03it/s]
Validation DataLoader 0: 41%|████ | 129/313 [00:00<00:01, 162.02it/s]
Validation DataLoader 0: 42%|████▏ | 130/313 [00:00<00:01, 162.00it/s]
Validation DataLoader 0: 42%|████▏ | 131/313 [00:00<00:01, 162.00it/s]
Validation DataLoader 0: 42%|████▏ | 132/313 [00:00<00:01, 162.04it/s]
Validation DataLoader 0: 42%|████▏ | 133/313 [00:00<00:01, 162.03it/s]
Validation DataLoader 0: 43%|████▎ | 134/313 [00:00<00:01, 162.07it/s]
Validation DataLoader 0: 43%|████▎ | 135/313 [00:00<00:01, 162.12it/s]
Validation DataLoader 0: 43%|████▎ | 136/313 [00:00<00:01, 162.15it/s]
Validation DataLoader 0: 44%|████▍ | 137/313 [00:00<00:01, 162.18it/s]
Validation DataLoader 0: 44%|████▍ | 138/313 [00:00<00:01, 162.24it/s]
Validation DataLoader 0: 44%|████▍ | 139/313 [00:00<00:01, 162.26it/s]
Validation DataLoader 0: 45%|████▍ | 140/313 [00:00<00:01, 162.31it/s]
Validation DataLoader 0: 45%|████▌ | 141/313 [00:00<00:01, 162.31it/s]
Validation DataLoader 0: 45%|████▌ | 142/313 [00:00<00:01, 162.33it/s]
Validation DataLoader 0: 46%|████▌ | 143/313 [00:00<00:01, 162.35it/s]
Validation DataLoader 0: 46%|████▌ | 144/313 [00:00<00:01, 162.36it/s]
Validation DataLoader 0: 46%|████▋ | 145/313 [00:00<00:01, 162.37it/s]
Validation DataLoader 0: 47%|████▋ | 146/313 [00:00<00:01, 162.38it/s]
Validation DataLoader 0: 47%|████▋ | 147/313 [00:00<00:01, 162.41it/s]
Validation DataLoader 0: 47%|████▋ | 148/313 [00:00<00:01, 162.44it/s]
Validation DataLoader 0: 48%|████▊ | 149/313 [00:00<00:01, 162.48it/s]
Validation DataLoader 0: 48%|████▊ | 150/313 [00:00<00:01, 162.49it/s]
Validation DataLoader 0: 48%|████▊ | 151/313 [00:00<00:00, 162.53it/s]
Validation DataLoader 0: 49%|████▊ | 152/313 [00:00<00:00, 162.54it/s]
Validation DataLoader 0: 49%|████▉ | 153/313 [00:00<00:00, 162.58it/s]
Validation DataLoader 0: 49%|████▉ | 154/313 [00:00<00:00, 162.62it/s]
Validation DataLoader 0: 50%|████▉ | 155/313 [00:00<00:00, 162.63it/s]
Validation DataLoader 0: 50%|████▉ | 156/313 [00:00<00:00, 162.67it/s]
Validation DataLoader 0: 50%|█████ | 157/313 [00:00<00:00, 162.69it/s]
Validation DataLoader 0: 50%|█████ | 158/313 [00:00<00:00, 162.73it/s]
Validation DataLoader 0: 51%|█████ | 159/313 [00:00<00:00, 162.76it/s]
Validation DataLoader 0: 51%|█████ | 160/313 [00:00<00:00, 162.79it/s]
Validation DataLoader 0: 51%|█████▏ | 161/313 [00:00<00:00, 162.81it/s]
Validation DataLoader 0: 52%|█████▏ | 162/313 [00:00<00:00, 162.83it/s]
Validation DataLoader 0: 52%|█████▏ | 163/313 [00:01<00:00, 162.85it/s]
Validation DataLoader 0: 52%|█████▏ | 164/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 53%|█████▎ | 165/313 [00:01<00:00, 162.90it/s]
Validation DataLoader 0: 53%|█████▎ | 166/313 [00:01<00:00, 162.91it/s]
Validation DataLoader 0: 53%|█████▎ | 167/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 54%|█████▎ | 168/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 54%|█████▍ | 169/313 [00:01<00:00, 162.93it/s]
Validation DataLoader 0: 54%|█████▍ | 170/313 [00:01<00:00, 162.96it/s]
Validation DataLoader 0: 55%|█████▍ | 171/313 [00:01<00:00, 162.99it/s]
Validation DataLoader 0: 55%|█████▍ | 172/313 [00:01<00:00, 163.02it/s]
Validation DataLoader 0: 55%|█████▌ | 173/313 [00:01<00:00, 163.03it/s]
Validation DataLoader 0: 56%|█████▌ | 174/313 [00:01<00:00, 163.05it/s]
Validation DataLoader 0: 56%|█████▌ | 175/313 [00:01<00:00, 163.09it/s]
Validation DataLoader 0: 56%|█████▌ | 176/313 [00:01<00:00, 163.11it/s]
Validation DataLoader 0: 57%|█████▋ | 177/313 [00:01<00:00, 163.14it/s]
Validation DataLoader 0: 57%|█████▋ | 178/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 57%|█████▋ | 179/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 58%|█████▊ | 180/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 58%|█████▊ | 181/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 58%|█████▊ | 182/313 [00:01<00:00, 162.93it/s]
Validation DataLoader 0: 58%|█████▊ | 183/313 [00:01<00:00, 162.94it/s]
Validation DataLoader 0: 59%|█████▉ | 184/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 59%|█████▉ | 185/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 59%|█████▉ | 186/313 [00:01<00:00, 162.99it/s]
Validation DataLoader 0: 60%|█████▉ | 187/313 [00:01<00:00, 163.01it/s]
Validation DataLoader 0: 60%|██████ | 188/313 [00:01<00:00, 163.02it/s]
Validation DataLoader 0: 60%|██████ | 189/313 [00:01<00:00, 163.04it/s]
Validation DataLoader 0: 61%|██████ | 190/313 [00:01<00:00, 163.06it/s]
Validation DataLoader 0: 61%|██████ | 191/313 [00:01<00:00, 163.09it/s]
Validation DataLoader 0: 61%|██████▏ | 192/313 [00:01<00:00, 163.12it/s]
Validation DataLoader 0: 62%|██████▏ | 193/313 [00:01<00:00, 163.15it/s]
Validation DataLoader 0: 62%|██████▏ | 194/313 [00:01<00:00, 163.18it/s]
Validation DataLoader 0: 62%|██████▏ | 195/313 [00:01<00:00, 163.20it/s]
Validation DataLoader 0: 63%|██████▎ | 196/313 [00:01<00:00, 163.23it/s]
Validation DataLoader 0: 63%|██████▎ | 197/313 [00:01<00:00, 163.26it/s]
Validation DataLoader 0: 63%|██████▎ | 198/313 [00:01<00:00, 163.24it/s]
Validation DataLoader 0: 64%|██████▎ | 199/313 [00:01<00:00, 163.26it/s]
Validation DataLoader 0: 64%|██████▍ | 200/313 [00:01<00:00, 163.27it/s]
Validation DataLoader 0: 64%|██████▍ | 201/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 65%|██████▍ | 202/313 [00:01<00:00, 163.32it/s]
Validation DataLoader 0: 65%|██████▍ | 203/313 [00:01<00:00, 163.23it/s]
Validation DataLoader 0: 65%|██████▌ | 204/313 [00:01<00:00, 163.12it/s]
Validation DataLoader 0: 65%|██████▌ | 205/313 [00:01<00:00, 163.11it/s]
Validation DataLoader 0: 66%|██████▌ | 206/313 [00:01<00:00, 163.12it/s]
Validation DataLoader 0: 66%|██████▌ | 207/313 [00:01<00:00, 163.13it/s]
Validation DataLoader 0: 66%|██████▋ | 208/313 [00:01<00:00, 163.14it/s]
Validation DataLoader 0: 67%|██████▋ | 209/313 [00:01<00:00, 163.15it/s]
Validation DataLoader 0: 67%|██████▋ | 210/313 [00:01<00:00, 163.17it/s]
Validation DataLoader 0: 67%|██████▋ | 211/313 [00:01<00:00, 163.18it/s]
Validation DataLoader 0: 68%|██████▊ | 212/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 68%|██████▊ | 213/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 68%|██████▊ | 214/313 [00:01<00:00, 163.17it/s]
Validation DataLoader 0: 69%|██████▊ | 215/313 [00:01<00:00, 163.06it/s]
Validation DataLoader 0: 69%|██████▉ | 216/313 [00:01<00:00, 163.10it/s]
Validation DataLoader 0: 69%|██████▉ | 217/313 [00:01<00:00, 162.99it/s]
Validation DataLoader 0: 70%|██████▉ | 218/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 70%|██████▉ | 219/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 70%|███████ | 220/313 [00:01<00:00, 162.98it/s]
Validation DataLoader 0: 71%|███████ | 221/313 [00:01<00:00, 162.96it/s]
Validation DataLoader 0: 71%|███████ | 222/313 [00:01<00:00, 162.98it/s]
Validation DataLoader 0: 71%|███████ | 223/313 [00:01<00:00, 162.99it/s]
Validation DataLoader 0: 72%|███████▏ | 224/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 72%|███████▏ | 225/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 72%|███████▏ | 226/313 [00:01<00:00, 162.93it/s]
Validation DataLoader 0: 73%|███████▎ | 227/313 [00:01<00:00, 162.93it/s]
Validation DataLoader 0: 73%|███████▎ | 228/313 [00:01<00:00, 162.83it/s]
Validation DataLoader 0: 73%|███████▎ | 229/313 [00:01<00:00, 162.73it/s]
Validation DataLoader 0: 73%|███████▎ | 230/313 [00:01<00:00, 162.68it/s]
Validation DataLoader 0: 74%|███████▍ | 231/313 [00:01<00:00, 162.68it/s]
Validation DataLoader 0: 74%|███████▍ | 232/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 74%|███████▍ | 233/313 [00:01<00:00, 162.54it/s]
Validation DataLoader 0: 75%|███████▍ | 234/313 [00:01<00:00, 162.46it/s]
Validation DataLoader 0: 75%|███████▌ | 235/313 [00:01<00:00, 162.45it/s]
Validation DataLoader 0: 75%|███████▌ | 236/313 [00:01<00:00, 162.46it/s]
Validation DataLoader 0: 76%|███████▌ | 237/313 [00:01<00:00, 162.46it/s]
Validation DataLoader 0: 76%|███████▌ | 238/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 76%|███████▋ | 239/313 [00:01<00:00, 162.42it/s]
Validation DataLoader 0: 77%|███████▋ | 240/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 77%|███████▋ | 241/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 77%|███████▋ | 242/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 78%|███████▊ | 243/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 78%|███████▊ | 244/313 [00:01<00:00, 162.42it/s]
Validation DataLoader 0: 78%|███████▊ | 245/313 [00:01<00:00, 162.43it/s]
Validation DataLoader 0: 79%|███████▊ | 246/313 [00:01<00:00, 162.45it/s]
Validation DataLoader 0: 79%|███████▉ | 247/313 [00:01<00:00, 162.35it/s]
Validation DataLoader 0: 79%|███████▉ | 248/313 [00:01<00:00, 162.26it/s]
Validation DataLoader 0: 80%|███████▉ | 249/313 [00:01<00:00, 162.25it/s]
Validation DataLoader 0: 80%|███████▉ | 250/313 [00:01<00:00, 162.24it/s]
Validation DataLoader 0: 80%|████████ | 251/313 [00:01<00:00, 162.24it/s]
Validation DataLoader 0: 81%|████████ | 252/313 [00:01<00:00, 162.23it/s]
Validation DataLoader 0: 81%|████████ | 253/313 [00:01<00:00, 162.25it/s]
Validation DataLoader 0: 81%|████████ | 254/313 [00:01<00:00, 162.22it/s]
Validation DataLoader 0: 81%|████████▏ | 255/313 [00:01<00:00, 162.19it/s]
Validation DataLoader 0: 82%|████████▏ | 256/313 [00:01<00:00, 162.18it/s]
Validation DataLoader 0: 82%|████████▏ | 257/313 [00:01<00:00, 162.17it/s]
Validation DataLoader 0: 82%|████████▏ | 258/313 [00:01<00:00, 162.16it/s]
Validation DataLoader 0: 83%|████████▎ | 259/313 [00:01<00:00, 162.16it/s]
Validation DataLoader 0: 83%|████████▎ | 260/313 [00:01<00:00, 162.15it/s]
Validation DataLoader 0: 83%|████████▎ | 261/313 [00:01<00:00, 162.18it/s]
Validation DataLoader 0: 84%|████████▎ | 262/313 [00:01<00:00, 162.20it/s]
Validation DataLoader 0: 84%|████████▍ | 263/313 [00:01<00:00, 162.22it/s]
Validation DataLoader 0: 84%|████████▍ | 264/313 [00:01<00:00, 162.23it/s]
Validation DataLoader 0: 85%|████████▍ | 265/313 [00:01<00:00, 162.25it/s]
Validation DataLoader 0: 85%|████████▍ | 266/313 [00:01<00:00, 162.28it/s]
Validation DataLoader 0: 85%|████████▌ | 267/313 [00:01<00:00, 162.30it/s]
Validation DataLoader 0: 86%|████████▌ | 268/313 [00:01<00:00, 162.31it/s]
Validation DataLoader 0: 86%|████████▌ | 269/313 [00:01<00:00, 162.33it/s]
Validation DataLoader 0: 86%|████████▋ | 270/313 [00:01<00:00, 162.33it/s]
Validation DataLoader 0: 87%|████████▋ | 271/313 [00:01<00:00, 162.34it/s]
Validation DataLoader 0: 87%|████████▋ | 272/313 [00:01<00:00, 162.35it/s]
Validation DataLoader 0: 87%|████████▋ | 273/313 [00:01<00:00, 162.36it/s]
Validation DataLoader 0: 88%|████████▊ | 274/313 [00:01<00:00, 162.36it/s]
Validation DataLoader 0: 88%|████████▊ | 275/313 [00:01<00:00, 162.37it/s]
Validation DataLoader 0: 88%|████████▊ | 276/313 [00:01<00:00, 162.37it/s]
Validation DataLoader 0: 88%|████████▊ | 277/313 [00:01<00:00, 162.37it/s]
Validation DataLoader 0: 89%|████████▉ | 278/313 [00:01<00:00, 162.38it/s]
Validation DataLoader 0: 89%|████████▉ | 279/313 [00:01<00:00, 162.38it/s]
Validation DataLoader 0: 89%|████████▉ | 280/313 [00:01<00:00, 162.39it/s]
Validation DataLoader 0: 90%|████████▉ | 281/313 [00:01<00:00, 162.41it/s]
Validation DataLoader 0: 90%|█████████ | 282/313 [00:01<00:00, 162.40it/s]
Validation DataLoader 0: 90%|█████████ | 283/313 [00:01<00:00, 162.43it/s]
Validation DataLoader 0: 91%|█████████ | 284/313 [00:01<00:00, 162.44it/s]
Validation DataLoader 0: 91%|█████████ | 285/313 [00:01<00:00, 162.46it/s]
Validation DataLoader 0: 91%|█████████▏| 286/313 [00:01<00:00, 162.47it/s]
Validation DataLoader 0: 92%|█████████▏| 287/313 [00:01<00:00, 162.47it/s]
Validation DataLoader 0: 92%|█████████▏| 288/313 [00:01<00:00, 162.47it/s]
Validation DataLoader 0: 92%|█████████▏| 289/313 [00:01<00:00, 162.50it/s]
Validation DataLoader 0: 93%|█████████▎| 290/313 [00:01<00:00, 162.51it/s]
Validation DataLoader 0: 93%|█████████▎| 291/313 [00:01<00:00, 162.54it/s]
Validation DataLoader 0: 93%|█████████▎| 292/313 [00:01<00:00, 162.55it/s]
Validation DataLoader 0: 94%|█████████▎| 293/313 [00:01<00:00, 162.56it/s]
Validation DataLoader 0: 94%|█████████▍| 294/313 [00:01<00:00, 162.58it/s]
Validation DataLoader 0: 94%|█████████▍| 295/313 [00:01<00:00, 162.59it/s]
Validation DataLoader 0: 95%|█████████▍| 296/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 95%|█████████▍| 297/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 95%|█████████▌| 298/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 96%|█████████▌| 299/313 [00:01<00:00, 162.64it/s]
Validation DataLoader 0: 96%|█████████▌| 300/313 [00:01<00:00, 162.66it/s]
Validation DataLoader 0: 96%|█████████▌| 301/313 [00:01<00:00, 162.66it/s]
Validation DataLoader 0: 96%|█████████▋| 302/313 [00:01<00:00, 162.69it/s]
Validation DataLoader 0: 97%|█████████▋| 303/313 [00:01<00:00, 162.69it/s]
Validation DataLoader 0: 97%|█████████▋| 304/313 [00:01<00:00, 162.71it/s]
Validation DataLoader 0: 97%|█████████▋| 305/313 [00:01<00:00, 162.73it/s]
Validation DataLoader 0: 98%|█████████▊| 306/313 [00:01<00:00, 162.74it/s]
Validation DataLoader 0: 98%|█████████▊| 307/313 [00:01<00:00, 162.75it/s]
Validation DataLoader 0: 98%|█████████▊| 308/313 [00:01<00:00, 162.76it/s]
Validation DataLoader 0: 99%|█████████▊| 309/313 [00:01<00:00, 162.77it/s]
Validation DataLoader 0: 99%|█████████▉| 310/313 [00:01<00:00, 162.79it/s]
Validation DataLoader 0: 99%|█████████▉| 311/313 [00:01<00:00, 162.80it/s]
Validation DataLoader 0: 100%|█████████▉| 312/313 [00:01<00:00, 162.81it/s]
Validation DataLoader 0: 100%|██████████| 313/313 [00:01<00:00, 162.93it/s]
Epoch 3: 100%|██████████| 3/3 [00:01<00:00, 1.53it/s, v_num=fp9x]
→ creating new artifact version for key 'testmodels/wandb/1c84fp9x.ckpt' in storage '/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops'
Epoch 3: 100%|██████████| 3/3 [00:01<00:00, 1.50it/s, v_num=fp9x]
Epoch 3: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 4: 0%| | 0/3 [00:00<?, ?it/s, v_num=fp9x]
Epoch 4: 33%|███▎ | 1/3 [00:00<00:00, 129.87it/s, v_num=fp9x]
Epoch 4: 33%|███▎ | 1/3 [00:00<00:00, 120.76it/s, v_num=fp9x]
Epoch 4: 67%|██████▋ | 2/3 [00:00<00:00, 134.01it/s, v_num=fp9x]
Epoch 4: 67%|██████▋ | 2/3 [00:00<00:00, 128.89it/s, v_num=fp9x]
Epoch 4: 100%|██████████| 3/3 [00:00<00:00, 134.35it/s, v_num=fp9x]
Epoch 4: 100%|██████████| 3/3 [00:00<00:00, 131.20it/s, v_num=fp9x]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: | | 0/? [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/313 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 1/313 [00:00<00:01, 223.39it/s]
Validation DataLoader 0: 1%| | 2/313 [00:00<00:01, 195.79it/s]
Validation DataLoader 0: 1%| | 3/313 [00:00<00:01, 182.81it/s]
Validation DataLoader 0: 1%|▏ | 4/313 [00:00<00:01, 180.01it/s]
Validation DataLoader 0: 2%|▏ | 5/313 [00:00<00:01, 176.63it/s]
Validation DataLoader 0: 2%|▏ | 6/313 [00:00<00:01, 173.80it/s]
Validation DataLoader 0: 2%|▏ | 7/313 [00:00<00:01, 173.28it/s]
Validation DataLoader 0: 3%|▎ | 8/313 [00:00<00:01, 172.69it/s]
Validation DataLoader 0: 3%|▎ | 9/313 [00:00<00:01, 171.87it/s]
Validation DataLoader 0: 3%|▎ | 10/313 [00:00<00:01, 171.79it/s]
Validation DataLoader 0: 4%|▎ | 11/313 [00:00<00:01, 171.15it/s]
Validation DataLoader 0: 4%|▍ | 12/313 [00:00<00:01, 170.77it/s]
Validation DataLoader 0: 4%|▍ | 13/313 [00:00<00:01, 170.76it/s]
Validation DataLoader 0: 4%|▍ | 14/313 [00:00<00:01, 170.48it/s]
Validation DataLoader 0: 5%|▍ | 15/313 [00:00<00:01, 170.46it/s]
Validation DataLoader 0: 5%|▌ | 16/313 [00:00<00:01, 170.24it/s]
Validation DataLoader 0: 5%|▌ | 17/313 [00:00<00:01, 170.19it/s]
Validation DataLoader 0: 6%|▌ | 18/313 [00:00<00:01, 170.09it/s]
Validation DataLoader 0: 6%|▌ | 19/313 [00:00<00:01, 169.73it/s]
Validation DataLoader 0: 6%|▋ | 20/313 [00:00<00:01, 169.57it/s]
Validation DataLoader 0: 7%|▋ | 21/313 [00:00<00:01, 169.43it/s]
Validation DataLoader 0: 7%|▋ | 22/313 [00:00<00:01, 168.72it/s]
Validation DataLoader 0: 7%|▋ | 23/313 [00:00<00:01, 168.74it/s]
Validation DataLoader 0: 8%|▊ | 24/313 [00:00<00:01, 168.70it/s]
Validation DataLoader 0: 8%|▊ | 25/313 [00:00<00:01, 168.76it/s]
Validation DataLoader 0: 8%|▊ | 26/313 [00:00<00:01, 168.70it/s]
Validation DataLoader 0: 9%|▊ | 27/313 [00:00<00:01, 168.69it/s]
Validation DataLoader 0: 9%|▉ | 28/313 [00:00<00:01, 168.70it/s]
Validation DataLoader 0: 9%|▉ | 29/313 [00:00<00:01, 168.68it/s]
Validation DataLoader 0: 10%|▉ | 30/313 [00:00<00:01, 168.70it/s]
Validation DataLoader 0: 10%|▉ | 31/313 [00:00<00:01, 168.67it/s]
Validation DataLoader 0: 10%|█ | 32/313 [00:00<00:01, 168.63it/s]
Validation DataLoader 0: 11%|█ | 33/313 [00:00<00:01, 168.38it/s]
Validation DataLoader 0: 11%|█ | 34/313 [00:00<00:01, 168.38it/s]
Validation DataLoader 0: 11%|█ | 35/313 [00:00<00:01, 166.74it/s]
Validation DataLoader 0: 12%|█▏ | 36/313 [00:00<00:01, 166.74it/s]
Validation DataLoader 0: 12%|█▏ | 37/313 [00:00<00:01, 166.82it/s]
Validation DataLoader 0: 12%|█▏ | 38/313 [00:00<00:01, 166.82it/s]
Validation DataLoader 0: 12%|█▏ | 39/313 [00:00<00:01, 166.84it/s]
Validation DataLoader 0: 13%|█▎ | 40/313 [00:00<00:01, 166.84it/s]
Validation DataLoader 0: 13%|█▎ | 41/313 [00:00<00:01, 166.82it/s]
Validation DataLoader 0: 13%|█▎ | 42/313 [00:00<00:01, 166.90it/s]
Validation DataLoader 0: 14%|█▎ | 43/313 [00:00<00:01, 166.84it/s]
Validation DataLoader 0: 14%|█▍ | 44/313 [00:00<00:01, 166.90it/s]
Validation DataLoader 0: 14%|█▍ | 45/313 [00:00<00:01, 166.84it/s]
Validation DataLoader 0: 15%|█▍ | 46/313 [00:00<00:01, 166.97it/s]
Validation DataLoader 0: 15%|█▌ | 47/313 [00:00<00:01, 166.86it/s]
Validation DataLoader 0: 15%|█▌ | 48/313 [00:00<00:01, 166.89it/s]
Validation DataLoader 0: 16%|█▌ | 49/313 [00:00<00:01, 166.95it/s]
Validation DataLoader 0: 16%|█▌ | 50/313 [00:00<00:01, 166.99it/s]
Validation DataLoader 0: 16%|█▋ | 51/313 [00:00<00:01, 167.04it/s]
Validation DataLoader 0: 17%|█▋ | 52/313 [00:00<00:01, 166.91it/s]
Validation DataLoader 0: 17%|█▋ | 53/313 [00:00<00:01, 167.00it/s]
Validation DataLoader 0: 17%|█▋ | 54/313 [00:00<00:01, 167.10it/s]
Validation DataLoader 0: 18%|█▊ | 55/313 [00:00<00:01, 167.07it/s]
Validation DataLoader 0: 18%|█▊ | 56/313 [00:00<00:01, 167.08it/s]
Validation DataLoader 0: 18%|█▊ | 57/313 [00:00<00:01, 167.05it/s]
Validation DataLoader 0: 19%|█▊ | 58/313 [00:00<00:01, 167.09it/s]
Validation DataLoader 0: 19%|█▉ | 59/313 [00:00<00:01, 167.12it/s]
Validation DataLoader 0: 19%|█▉ | 60/313 [00:00<00:01, 167.06it/s]
Validation DataLoader 0: 19%|█▉ | 61/313 [00:00<00:01, 167.04it/s]
Validation DataLoader 0: 20%|█▉ | 62/313 [00:00<00:01, 167.06it/s]
Validation DataLoader 0: 20%|██ | 63/313 [00:00<00:01, 167.08it/s]
Validation DataLoader 0: 20%|██ | 64/313 [00:00<00:01, 166.93it/s]
Validation DataLoader 0: 21%|██ | 65/313 [00:00<00:01, 166.96it/s]
Validation DataLoader 0: 21%|██ | 66/313 [00:00<00:01, 167.06it/s]
Validation DataLoader 0: 21%|██▏ | 67/313 [00:00<00:01, 167.03it/s]
Validation DataLoader 0: 22%|██▏ | 68/313 [00:00<00:01, 166.84it/s]
Validation DataLoader 0: 22%|██▏ | 69/313 [00:00<00:01, 166.77it/s]
Validation DataLoader 0: 22%|██▏ | 70/313 [00:00<00:01, 166.77it/s]
Validation DataLoader 0: 23%|██▎ | 71/313 [00:00<00:01, 166.76it/s]
Validation DataLoader 0: 23%|██▎ | 72/313 [00:00<00:01, 166.73it/s]
Validation DataLoader 0: 23%|██▎ | 73/313 [00:00<00:01, 166.72it/s]
Validation DataLoader 0: 24%|██▎ | 74/313 [00:00<00:01, 166.69it/s]
Validation DataLoader 0: 24%|██▍ | 75/313 [00:00<00:01, 166.67it/s]
Validation DataLoader 0: 24%|██▍ | 76/313 [00:00<00:01, 166.66it/s]
Validation DataLoader 0: 25%|██▍ | 77/313 [00:00<00:01, 166.67it/s]
Validation DataLoader 0: 25%|██▍ | 78/313 [00:00<00:01, 166.69it/s]
Validation DataLoader 0: 25%|██▌ | 79/313 [00:00<00:01, 166.70it/s]
Validation DataLoader 0: 26%|██▌ | 80/313 [00:00<00:01, 166.64it/s]
Validation DataLoader 0: 26%|██▌ | 81/313 [00:00<00:01, 166.61it/s]
Validation DataLoader 0: 26%|██▌ | 82/313 [00:00<00:01, 166.60it/s]
Validation DataLoader 0: 27%|██▋ | 83/313 [00:00<00:01, 166.58it/s]
Validation DataLoader 0: 27%|██▋ | 84/313 [00:00<00:01, 166.58it/s]
Validation DataLoader 0: 27%|██▋ | 85/313 [00:00<00:01, 166.57it/s]
Validation DataLoader 0: 27%|██▋ | 86/313 [00:00<00:01, 166.58it/s]
Validation DataLoader 0: 28%|██▊ | 87/313 [00:00<00:01, 166.45it/s]
Validation DataLoader 0: 28%|██▊ | 88/313 [00:00<00:01, 166.37it/s]
Validation DataLoader 0: 28%|██▊ | 89/313 [00:00<00:01, 166.38it/s]
Validation DataLoader 0: 29%|██▉ | 90/313 [00:00<00:01, 166.40it/s]
Validation DataLoader 0: 29%|██▉ | 91/313 [00:00<00:01, 166.42it/s]
Validation DataLoader 0: 29%|██▉ | 92/313 [00:00<00:01, 166.45it/s]
Validation DataLoader 0: 30%|██▉ | 93/313 [00:00<00:01, 166.45it/s]
Validation DataLoader 0: 30%|███ | 94/313 [00:00<00:01, 166.47it/s]
Validation DataLoader 0: 30%|███ | 95/313 [00:00<00:01, 166.47it/s]
Validation DataLoader 0: 31%|███ | 96/313 [00:00<00:01, 166.50it/s]
Validation DataLoader 0: 31%|███ | 97/313 [00:00<00:01, 166.51it/s]
Validation DataLoader 0: 31%|███▏ | 98/313 [00:00<00:01, 166.49it/s]
Validation DataLoader 0: 32%|███▏ | 99/313 [00:00<00:01, 166.56it/s]
Validation DataLoader 0: 32%|███▏ | 100/313 [00:00<00:01, 166.56it/s]
Validation DataLoader 0: 32%|███▏ | 101/313 [00:00<00:01, 166.54it/s]
Validation DataLoader 0: 33%|███▎ | 102/313 [00:00<00:01, 166.55it/s]
Validation DataLoader 0: 33%|███▎ | 103/313 [00:00<00:01, 166.55it/s]
Validation DataLoader 0: 33%|███▎ | 104/313 [00:00<00:01, 166.56it/s]
Validation DataLoader 0: 34%|███▎ | 105/313 [00:00<00:01, 166.47it/s]
Validation DataLoader 0: 34%|███▍ | 106/313 [00:00<00:01, 166.47it/s]
Validation DataLoader 0: 34%|███▍ | 107/313 [00:00<00:01, 166.43it/s]
Validation DataLoader 0: 35%|███▍ | 108/313 [00:00<00:01, 166.40it/s]
Validation DataLoader 0: 35%|███▍ | 109/313 [00:00<00:01, 166.41it/s]
Validation DataLoader 0: 35%|███▌ | 110/313 [00:00<00:01, 166.43it/s]
Validation DataLoader 0: 35%|███▌ | 111/313 [00:00<00:01, 166.41it/s]
Validation DataLoader 0: 36%|███▌ | 112/313 [00:00<00:01, 166.44it/s]
Validation DataLoader 0: 36%|███▌ | 113/313 [00:00<00:01, 166.45it/s]
Validation DataLoader 0: 36%|███▋ | 114/313 [00:00<00:01, 166.34it/s]
Validation DataLoader 0: 37%|███▋ | 115/313 [00:00<00:01, 166.31it/s]
Validation DataLoader 0: 37%|███▋ | 116/313 [00:00<00:01, 166.29it/s]
Validation DataLoader 0: 37%|███▋ | 117/313 [00:00<00:01, 166.32it/s]
Validation DataLoader 0: 38%|███▊ | 118/313 [00:00<00:01, 166.30it/s]
Validation DataLoader 0: 38%|███▊ | 119/313 [00:00<00:01, 166.34it/s]
Validation DataLoader 0: 38%|███▊ | 120/313 [00:00<00:01, 166.36it/s]
Validation DataLoader 0: 39%|███▊ | 121/313 [00:00<00:01, 166.36it/s]
Validation DataLoader 0: 39%|███▉ | 122/313 [00:00<00:01, 166.34it/s]
Validation DataLoader 0: 39%|███▉ | 123/313 [00:00<00:01, 166.32it/s]
Validation DataLoader 0: 40%|███▉ | 124/313 [00:00<00:01, 166.28it/s]
Validation DataLoader 0: 40%|███▉ | 125/313 [00:00<00:01, 166.31it/s]
Validation DataLoader 0: 40%|████ | 126/313 [00:00<00:01, 166.31it/s]
Validation DataLoader 0: 41%|████ | 127/313 [00:00<00:01, 166.31it/s]
Validation DataLoader 0: 41%|████ | 128/313 [00:00<00:01, 166.33it/s]
Validation DataLoader 0: 41%|████ | 129/313 [00:00<00:01, 166.30it/s]
Validation DataLoader 0: 42%|████▏ | 130/313 [00:00<00:01, 166.05it/s]
Validation DataLoader 0: 42%|████▏ | 131/313 [00:00<00:01, 165.84it/s]
Validation DataLoader 0: 42%|████▏ | 132/313 [00:00<00:01, 165.69it/s]
Validation DataLoader 0: 42%|████▏ | 133/313 [00:00<00:01, 165.51it/s]
Validation DataLoader 0: 43%|████▎ | 134/313 [00:00<00:01, 165.42it/s]
Validation DataLoader 0: 43%|████▎ | 135/313 [00:00<00:01, 165.35it/s]
Validation DataLoader 0: 43%|████▎ | 136/313 [00:00<00:01, 165.23it/s]
Validation DataLoader 0: 44%|████▍ | 137/313 [00:00<00:01, 165.07it/s]
Validation DataLoader 0: 44%|████▍ | 138/313 [00:00<00:01, 164.87it/s]
Validation DataLoader 0: 44%|████▍ | 139/313 [00:00<00:01, 164.81it/s]
Validation DataLoader 0: 45%|████▍ | 140/313 [00:00<00:01, 164.69it/s]
Validation DataLoader 0: 45%|████▌ | 141/313 [00:00<00:01, 164.52it/s]
Validation DataLoader 0: 45%|████▌ | 142/313 [00:00<00:01, 164.49it/s]
Validation DataLoader 0: 46%|████▌ | 143/313 [00:00<00:01, 164.48it/s]
Validation DataLoader 0: 46%|████▌ | 144/313 [00:00<00:01, 164.30it/s]
Validation DataLoader 0: 46%|████▋ | 145/313 [00:00<00:01, 164.24it/s]
Validation DataLoader 0: 47%|████▋ | 146/313 [00:00<00:01, 164.18it/s]
Validation DataLoader 0: 47%|████▋ | 147/313 [00:00<00:01, 164.14it/s]
Validation DataLoader 0: 47%|████▋ | 148/313 [00:00<00:01, 164.02it/s]
Validation DataLoader 0: 48%|████▊ | 149/313 [00:00<00:01, 163.81it/s]
Validation DataLoader 0: 48%|████▊ | 150/313 [00:00<00:00, 163.70it/s]
Validation DataLoader 0: 48%|████▊ | 151/313 [00:00<00:00, 163.56it/s]
Validation DataLoader 0: 49%|████▊ | 152/313 [00:00<00:00, 163.56it/s]
Validation DataLoader 0: 49%|████▉ | 153/313 [00:00<00:00, 163.43it/s]
Validation DataLoader 0: 49%|████▉ | 154/313 [00:00<00:00, 163.38it/s]
Validation DataLoader 0: 50%|████▉ | 155/313 [00:00<00:00, 163.27it/s]
Validation DataLoader 0: 50%|████▉ | 156/313 [00:00<00:00, 163.12it/s]
Validation DataLoader 0: 50%|█████ | 157/313 [00:00<00:00, 162.98it/s]
Validation DataLoader 0: 50%|█████ | 158/313 [00:00<00:00, 162.81it/s]
Validation DataLoader 0: 51%|█████ | 159/313 [00:00<00:00, 162.81it/s]
Validation DataLoader 0: 51%|█████ | 160/313 [00:00<00:00, 162.76it/s]
Validation DataLoader 0: 51%|█████▏ | 161/313 [00:00<00:00, 162.75it/s]
Validation DataLoader 0: 52%|█████▏ | 162/313 [00:00<00:00, 162.76it/s]
Validation DataLoader 0: 52%|█████▏ | 163/313 [00:01<00:00, 162.73it/s]
Validation DataLoader 0: 52%|█████▏ | 164/313 [00:01<00:00, 162.74it/s]
Validation DataLoader 0: 53%|█████▎ | 165/313 [00:01<00:00, 162.71it/s]
Validation DataLoader 0: 53%|█████▎ | 166/313 [00:01<00:00, 162.69it/s]
Validation DataLoader 0: 53%|█████▎ | 167/313 [00:01<00:00, 162.66it/s]
Validation DataLoader 0: 54%|█████▎ | 168/313 [00:01<00:00, 162.62it/s]
Validation DataLoader 0: 54%|█████▍ | 169/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 54%|█████▍ | 170/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 55%|█████▍ | 171/313 [00:01<00:00, 162.61it/s]
Validation DataLoader 0: 55%|█████▍ | 172/313 [00:01<00:00, 162.65it/s]
Validation DataLoader 0: 55%|█████▌ | 173/313 [00:01<00:00, 162.64it/s]
Validation DataLoader 0: 56%|█████▌ | 174/313 [00:01<00:00, 162.66it/s]
Validation DataLoader 0: 56%|█████▌ | 175/313 [00:01<00:00, 162.67it/s]
Validation DataLoader 0: 56%|█████▌ | 176/313 [00:01<00:00, 162.70it/s]
Validation DataLoader 0: 57%|█████▋ | 177/313 [00:01<00:00, 162.75it/s]
Validation DataLoader 0: 57%|█████▋ | 178/313 [00:01<00:00, 162.76it/s]
Validation DataLoader 0: 57%|█████▋ | 179/313 [00:01<00:00, 162.78it/s]
Validation DataLoader 0: 58%|█████▊ | 180/313 [00:01<00:00, 162.78it/s]
Validation DataLoader 0: 58%|█████▊ | 181/313 [00:01<00:00, 162.77it/s]
Validation DataLoader 0: 58%|█████▊ | 182/313 [00:01<00:00, 162.80it/s]
Validation DataLoader 0: 58%|█████▊ | 183/313 [00:01<00:00, 162.84it/s]
Validation DataLoader 0: 59%|█████▉ | 184/313 [00:01<00:00, 162.84it/s]
Validation DataLoader 0: 59%|█████▉ | 185/313 [00:01<00:00, 162.86it/s]
Validation DataLoader 0: 59%|█████▉ | 186/313 [00:01<00:00, 162.80it/s]
Validation DataLoader 0: 60%|█████▉ | 187/313 [00:01<00:00, 162.85it/s]
Validation DataLoader 0: 60%|██████ | 188/313 [00:01<00:00, 162.86it/s]
Validation DataLoader 0: 60%|██████ | 189/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 61%|██████ | 190/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 61%|██████ | 191/313 [00:01<00:00, 162.91it/s]
Validation DataLoader 0: 61%|██████▏ | 192/313 [00:01<00:00, 162.93it/s]
Validation DataLoader 0: 62%|██████▏ | 193/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 62%|██████▏ | 194/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 62%|██████▏ | 195/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 63%|██████▎ | 196/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 63%|██████▎ | 197/313 [00:01<00:00, 162.99it/s]
Validation DataLoader 0: 63%|██████▎ | 198/313 [00:01<00:00, 163.01it/s]
Validation DataLoader 0: 64%|██████▎ | 199/313 [00:01<00:00, 163.05it/s]
Validation DataLoader 0: 64%|██████▍ | 200/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 64%|██████▍ | 201/313 [00:01<00:00, 163.09it/s]
Validation DataLoader 0: 65%|██████▍ | 202/313 [00:01<00:00, 163.11it/s]
Validation DataLoader 0: 65%|██████▍ | 203/313 [00:01<00:00, 163.13it/s]
Validation DataLoader 0: 65%|██████▌ | 204/313 [00:01<00:00, 163.15it/s]
Validation DataLoader 0: 65%|██████▌ | 205/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 66%|██████▌ | 206/313 [00:01<00:00, 163.19it/s]
Validation DataLoader 0: 66%|██████▌ | 207/313 [00:01<00:00, 163.19it/s]
Validation DataLoader 0: 66%|██████▋ | 208/313 [00:01<00:00, 163.19it/s]
Validation DataLoader 0: 67%|██████▋ | 209/313 [00:01<00:00, 163.22it/s]
Validation DataLoader 0: 67%|██████▋ | 210/313 [00:01<00:00, 163.19it/s]
Validation DataLoader 0: 67%|██████▋ | 211/313 [00:01<00:00, 163.22it/s]
Validation DataLoader 0: 68%|██████▊ | 212/313 [00:01<00:00, 163.24it/s]
Validation DataLoader 0: 68%|██████▊ | 213/313 [00:01<00:00, 163.24it/s]
Validation DataLoader 0: 68%|██████▊ | 214/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 69%|██████▊ | 215/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 69%|██████▉ | 216/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 69%|██████▉ | 217/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 70%|██████▉ | 218/313 [00:01<00:00, 162.88it/s]
Validation DataLoader 0: 70%|██████▉ | 219/313 [00:01<00:00, 162.89it/s]
Validation DataLoader 0: 70%|███████ | 220/313 [00:01<00:00, 162.87it/s]
Validation DataLoader 0: 71%|███████ | 221/313 [00:01<00:00, 162.87it/s]
Validation DataLoader 0: 71%|███████ | 222/313 [00:01<00:00, 162.87it/s]
Validation DataLoader 0: 71%|███████ | 223/313 [00:01<00:00, 162.86it/s]
Validation DataLoader 0: 72%|███████▏ | 224/313 [00:01<00:00, 162.87it/s]
Validation DataLoader 0: 72%|███████▏ | 225/313 [00:01<00:00, 162.90it/s]
Validation DataLoader 0: 72%|███████▏ | 226/313 [00:01<00:00, 162.92it/s]
Validation DataLoader 0: 73%|███████▎ | 227/313 [00:01<00:00, 162.94it/s]
Validation DataLoader 0: 73%|███████▎ | 228/313 [00:01<00:00, 162.95it/s]
Validation DataLoader 0: 73%|███████▎ | 229/313 [00:01<00:00, 162.97it/s]
Validation DataLoader 0: 73%|███████▎ | 230/313 [00:01<00:00, 162.98it/s]
Validation DataLoader 0: 74%|███████▍ | 231/313 [00:01<00:00, 163.00it/s]
Validation DataLoader 0: 74%|███████▍ | 232/313 [00:01<00:00, 163.01it/s]
Validation DataLoader 0: 74%|███████▍ | 233/313 [00:01<00:00, 163.02it/s]
Validation DataLoader 0: 75%|███████▍ | 234/313 [00:01<00:00, 163.03it/s]
Validation DataLoader 0: 75%|███████▌ | 235/313 [00:01<00:00, 163.05it/s]
Validation DataLoader 0: 75%|███████▌ | 236/313 [00:01<00:00, 163.04it/s]
Validation DataLoader 0: 76%|███████▌ | 237/313 [00:01<00:00, 163.06it/s]
Validation DataLoader 0: 76%|███████▌ | 238/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 76%|███████▋ | 239/313 [00:01<00:00, 163.07it/s]
Validation DataLoader 0: 77%|███████▋ | 240/313 [00:01<00:00, 163.08it/s]
Validation DataLoader 0: 77%|███████▋ | 241/313 [00:01<00:00, 163.08it/s]
Validation DataLoader 0: 77%|███████▋ | 242/313 [00:01<00:00, 163.09it/s]
Validation DataLoader 0: 78%|███████▊ | 243/313 [00:01<00:00, 163.08it/s]
Validation DataLoader 0: 78%|███████▊ | 244/313 [00:01<00:00, 163.09it/s]
Validation DataLoader 0: 78%|███████▊ | 245/313 [00:01<00:00, 163.10it/s]
Validation DataLoader 0: 79%|███████▊ | 246/313 [00:01<00:00, 163.10it/s]
Validation DataLoader 0: 79%|███████▉ | 247/313 [00:01<00:00, 163.11it/s]
Validation DataLoader 0: 79%|███████▉ | 248/313 [00:01<00:00, 163.12it/s]
Validation DataLoader 0: 80%|███████▉ | 249/313 [00:01<00:00, 163.14it/s]
Validation DataLoader 0: 80%|███████▉ | 250/313 [00:01<00:00, 163.16it/s]
Validation DataLoader 0: 80%|████████ | 251/313 [00:01<00:00, 163.17it/s]
Validation DataLoader 0: 81%|████████ | 252/313 [00:01<00:00, 163.19it/s]
Validation DataLoader 0: 81%|████████ | 253/313 [00:01<00:00, 163.21it/s]
Validation DataLoader 0: 81%|████████ | 254/313 [00:01<00:00, 163.23it/s]
Validation DataLoader 0: 81%|████████▏ | 255/313 [00:01<00:00, 163.25it/s]
Validation DataLoader 0: 82%|████████▏ | 256/313 [00:01<00:00, 163.26it/s]
Validation DataLoader 0: 82%|████████▏ | 257/313 [00:01<00:00, 163.28it/s]
Validation DataLoader 0: 82%|████████▏ | 258/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 83%|████████▎ | 259/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 83%|████████▎ | 260/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 83%|████████▎ | 261/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 84%|████████▎ | 262/313 [00:01<00:00, 163.31it/s]
Validation DataLoader 0: 84%|████████▍ | 263/313 [00:01<00:00, 163.33it/s]
Validation DataLoader 0: 84%|████████▍ | 264/313 [00:01<00:00, 163.34it/s]
Validation DataLoader 0: 85%|████████▍ | 265/313 [00:01<00:00, 163.35it/s]
Validation DataLoader 0: 85%|████████▍ | 266/313 [00:01<00:00, 163.36it/s]
Validation DataLoader 0: 85%|████████▌ | 267/313 [00:01<00:00, 163.28it/s]
Validation DataLoader 0: 86%|████████▌ | 268/313 [00:01<00:00, 163.28it/s]
Validation DataLoader 0: 86%|████████▌ | 269/313 [00:01<00:00, 163.29it/s]
Validation DataLoader 0: 86%|████████▋ | 270/313 [00:01<00:00, 163.30it/s]
Validation DataLoader 0: 87%|████████▋ | 271/313 [00:01<00:00, 163.31it/s]
Validation DataLoader 0: 87%|████████▋ | 272/313 [00:01<00:00, 163.33it/s]
Validation DataLoader 0: 87%|████████▋ | 273/313 [00:01<00:00, 163.34it/s]
Validation DataLoader 0: 88%|████████▊ | 274/313 [00:01<00:00, 163.35it/s]
Validation DataLoader 0: 88%|████████▊ | 275/313 [00:01<00:00, 163.37it/s]
Validation DataLoader 0: 88%|████████▊ | 276/313 [00:01<00:00, 163.38it/s]
Validation DataLoader 0: 88%|████████▊ | 277/313 [00:01<00:00, 163.41it/s]
Validation DataLoader 0: 89%|████████▉ | 278/313 [00:01<00:00, 163.42it/s]
Validation DataLoader 0: 89%|████████▉ | 279/313 [00:01<00:00, 163.44it/s]
Validation DataLoader 0: 89%|████████▉ | 280/313 [00:01<00:00, 163.45it/s]
Validation DataLoader 0: 90%|████████▉ | 281/313 [00:01<00:00, 163.47it/s]
Validation DataLoader 0: 90%|█████████ | 282/313 [00:01<00:00, 163.48it/s]
Validation DataLoader 0: 90%|█████████ | 283/313 [00:01<00:00, 163.48it/s]
Validation DataLoader 0: 91%|█████████ | 284/313 [00:01<00:00, 163.50it/s]
Validation DataLoader 0: 91%|█████████ | 285/313 [00:01<00:00, 163.49it/s]
Validation DataLoader 0: 91%|█████████▏| 286/313 [00:01<00:00, 163.51it/s]
Validation DataLoader 0: 92%|█████████▏| 287/313 [00:01<00:00, 163.52it/s]
Validation DataLoader 0: 92%|█████████▏| 288/313 [00:01<00:00, 163.53it/s]
Validation DataLoader 0: 92%|█████████▏| 289/313 [00:01<00:00, 163.54it/s]
Validation DataLoader 0: 93%|█████████▎| 290/313 [00:01<00:00, 163.54it/s]
Validation DataLoader 0: 93%|█████████▎| 291/313 [00:01<00:00, 163.56it/s]
Validation DataLoader 0: 93%|█████████▎| 292/313 [00:01<00:00, 163.57it/s]
Validation DataLoader 0: 94%|█████████▎| 293/313 [00:01<00:00, 163.58it/s]
Validation DataLoader 0: 94%|█████████▍| 294/313 [00:01<00:00, 163.60it/s]
Validation DataLoader 0: 94%|█████████▍| 295/313 [00:01<00:00, 163.60it/s]
Validation DataLoader 0: 95%|█████████▍| 296/313 [00:01<00:00, 163.61it/s]
Validation DataLoader 0: 95%|█████████▍| 297/313 [00:01<00:00, 163.62it/s]
Validation DataLoader 0: 95%|█████████▌| 298/313 [00:01<00:00, 163.61it/s]
Validation DataLoader 0: 96%|█████████▌| 299/313 [00:01<00:00, 163.63it/s]
Validation DataLoader 0: 96%|█████████▌| 300/313 [00:01<00:00, 163.61it/s]
Validation DataLoader 0: 96%|█████████▌| 301/313 [00:01<00:00, 163.63it/s]
Validation DataLoader 0: 96%|█████████▋| 302/313 [00:01<00:00, 163.63it/s]
Validation DataLoader 0: 97%|█████████▋| 303/313 [00:01<00:00, 163.63it/s]
Validation DataLoader 0: 97%|█████████▋| 304/313 [00:01<00:00, 163.64it/s]
Validation DataLoader 0: 97%|█████████▋| 305/313 [00:01<00:00, 163.64it/s]
Validation DataLoader 0: 98%|█████████▊| 306/313 [00:01<00:00, 163.64it/s]
Validation DataLoader 0: 98%|█████████▊| 307/313 [00:01<00:00, 163.65it/s]
Validation DataLoader 0: 98%|█████████▊| 308/313 [00:01<00:00, 163.65it/s]
Validation DataLoader 0: 99%|█████████▊| 309/313 [00:01<00:00, 163.66it/s]
Validation DataLoader 0: 99%|█████████▉| 310/313 [00:01<00:00, 163.67it/s]
Validation DataLoader 0: 99%|█████████▉| 311/313 [00:01<00:00, 163.68it/s]
Validation DataLoader 0: 100%|█████████▉| 312/313 [00:01<00:00, 163.70it/s]
Validation DataLoader 0: 100%|██████████| 313/313 [00:01<00:00, 163.83it/s]
Epoch 4: 100%|██████████| 3/3 [00:01<00:00, 1.54it/s, v_num=fp9x]
→ creating new artifact version for key 'testmodels/wandb/1c84fp9x.ckpt' in storage '/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops'
Epoch 4: 100%|██████████| 3/3 [00:01<00:00, 1.51it/s, v_num=fp9x]
`Trainer.fit` stopped: `max_epochs=5` reached.
Epoch 4: 100%|██████████| 3/3 [00:01<00:00, 1.51it/s, v_num=fp9x]
wandb_logger.experiment.name
Show code cell output
'ghastly-cobweb-325'
wandb.finish()
Show code cell output
wandb: updating run metadata
wandb: uploading history steps 0-9, summary
wandb:
wandb: Run history:
wandb: epoch ▁▁▃▃▅▅▆▆██
wandb: train_loss_epoch █▆▄▃▁
wandb: trainer/global_step ▁▁▃▃▅▅▆▆██
wandb: val_loss █▆▅▃▁
wandb:
wandb: Run summary:
wandb: epoch 4
wandb: train_loss_epoch 0.11109
wandb: trainer/global_step 14
wandb: val_loss 0.11398
wandb:
wandb: 🚀 View run ghastly-cobweb-325 at: https://wandb.ai/lamin-mlops-demo/lamin/runs/1c84fp9x
wandb: ⭐️ View project at: https://wandb.ai/lamin-mlops-demo/lamin
wandb: Synced 4 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20251030_075850-1c84fp9x/logs
W&B and LaminDB user interfaces together¶
W&B and LaminDB runs:
Both W&B and LaminDB capture any runs together with run parameters.
W&B experiment overview |
LaminHub run overview |
|---|---|
|
|
W&B run details and LaminDB artifact details:
W&B and LaminDB complement each other. Whereas W&B is excellent at capturing metrics over time, LaminDB excells at capturing lineage of input & output data and training checkpoints.
W&B run view |
LaminHub run view |
|---|---|
|
|
Both frameworks display output artifacts that were generated during the run. LaminDB further captures input artifacts, their origin and the associated source code.
W&B artifact view |
LaminHub artifact view |
|---|---|
|
|
All checkpoints are automatically annotated by the specified training metrics and W&B run ID & name to keep both frameworks in sync:
last_checkpoint_af = ln.Artifact.filter(
key__startswith="testmodels/wandb/", suffix__endswith="ckpt", is_latest=True
).last()
last_checkpoint_af.describe()
Show code cell output
Artifact: testmodels/wandb/1c84fp9x.ckpt (0004) ├── uid: xZpz3Zk2x03877Vv0004 run: snk8iVl (wandb.ipynb) │ kind: model otype: None │ hash: XXdbfXff8PVH_oUS_S12NA size: 621.7 KB │ branch: main space: all │ created_at: 2025-10-30 07:59:01 UTC created_by: anonymous ├── storage/path: /home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops/.lamindb/xZpz3Zk2x03877Vv0004.ckpt ├── Features │ └── current_epoch int 4 │ train_loss float 0.11109409481287003 │ val_loss float 0.11397749930620193 │ wandb_run_id str 1c84fp9x │ wandb_run_name str ghastly-cobweb-325 └── Labels └── .projects Project Wandb tutorial
To reuse the checkpoint later:
last_checkpoint_af.cache()
Show code cell output
PosixUPath('/home/runner/work/lamin-mlops/lamin-mlops/docs/lamin-mlops/.lamindb/xZpz3Zk2x03877Vv0004.ckpt')
last_checkpoint_af.view_lineage()
Show code cell output
ln.finish()
Show code cell output
! cells [(11, 13)] were not run consecutively
→ finished Run('snk8iVlobFFFWEjY') after 16s at 2025-10-30 07:59:03 UTC