Releasing Blacksmith 1A

After nearly 3 months of training it has finally finished.

Blacksmith 1A is our first "real" LLM, with a size of 522M (0.5B) parameters and a context window of 128K (64K base; 128K at inference time). It has the same architecture as NanoGPT, with 20 transformer blocks, 10 attention heads and a model width of 1280. The base model is trained with 4K, then a series of long-context training is applied to lengthen it to 8K → 16K → 32K → 64K; then a supervised fine-tuning phase is adopted to turn it into a proper chatbot.

One'd say 0.5B is barely anything, and they'll be correct, and even among the 0.5B tier it probably leans more towards the "useless" end. The 128K claim is actually very flimsy: it technically can handle 128K, but the quality is somewhere between non-existent and not-enough - it can't remember things well unless it's very close to the prompt. I would like to have something that's at least around 3B, but that will cost years on my training machine, which would render the effort completely meaningless. Training models from scratch is largely a losing deal unless you're already in this kind of business; normal people without a lofty budget are much better to fine-tune an existing one like the smaller Qwen ones.

The run wasn't completely without merit, however. I at least have learned a lesson or two. At this small of a scale, which datasets you choose (for both pre-training and fine-tuning) and how you fine-tune it seems to be of utmost importance. When I started planning for Blacksmith 1 in late July, my goal was to turn it into a simple chatbot, for which the LLM I was using suggested UltraChat; this turns out to be more than enough to teach the chat format and pollute the model's answers with the common "as a language model I can't" and "I don't have personal preferences" phrases even for simple knowledge-only questions. Catastrophic forgetting, of course, is also always a problem. It's probably a good choice as well to actually train it with a bit more data than required for the so-called "chinchilla optimum" of ~20 tok/param, which I didn't do for Blacksmith 1A due to a plethora of resource limitations.

Would there be a Blacksmith 1.1 or Blacksmith 2? It largely comes down to whether I'd still have a job in 2027; if job, then yes. I probably won't do another dense one though, not until I've got my hands on a proper GPU. If there's a next version it would probably be a MoE one somewhat specialized for certain purposes like tool call or text summarization.

The model can be found on HuggingFace: TriusAI/blacksmith-1a. We've prepared a 4-bit quantized version as well.

— Z Lin