Skip to content

Add local baseline reproduction record#346

Open
bjbjbjbjbjbj wants to merge 5 commits intoopenai:mainfrom
bjbjbjbjbjbj:bj-local-baseline-pr
Open

Add local baseline reproduction record#346
bjbjbjbjbjbj wants to merge 5 commits intoopenai:mainfrom
bjbjbjbjbjbj:bj-local-baseline-pr

Conversation

@bjbjbjbjbjbj
Copy link

Summary

This PR adds a non-record local baseline reproduction run.

Environment

  • local single-GPU run
  • train_shards=1
  • seq_len=1024
  • grad_accum_steps=8

Best observed result

  • val_bpb = 1.3529 @ step 4200

Notes

  • validation improved from 4.1077 to 1.3529
  • performance plateaued after around step 4200
  • this is a local baseline reproduction, not a SOTA submission

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant