Login
You're viewing the social.telemetrydeck.com public feed.
  • Brandon Rohrerbrohrer@recsys.social
    Oct 18, 2026, 7:39 PM

    For the handful of you who are following along this journey, the artisanal language model I'm building to do proofreading is getting more mature.

    brandonrohrer.org/alms_blended

    One big step forward is the addition of additive smoothing (alpha) and upward-biased additive smoothing (beta) parameters. These helped to avoid false detections and increase precision - an important precursor to building ensemble models.

    Frequentist estimate of the conditional probability of the bigram AB, given the occurrence of the unigram A, expressed as P(AB | A). It is the total count of occurrences of AB, divided by the count of occurrences of A in any context. The other parameters are tweaks introduced to improve results on a proofreading task. k is the alphabet size of the tokenizer. alpha is a classic additive smoothing parameter. beta is variant of additive smoothing the introduces an upward bias for small sample sizes, preventing false detections.
    💬 1🔄 0⭐ 0

Replies

  • Brandon Rohrerbrohrer@recsys.social
    Oct 18, 2026, 7:39 PM

    The other big step was choosing a per-model optimization criterion, F-rho. This allowed a two-phase hyperparameter optimization approach where individual models were tuned before being combined into an ensemble. That leaves fewer free hyperparameters in the ensemble, making that secondary tuning process more practicable.

    A common optimization criterion is the [F-score](https://en.wikipedia.org/wiki/F-score). It's most common manifestation is the F1: F1 = 2 × precision × recall /        precision + recall The product over the sum of precision and recall gives a single metric that balances the concerns of them both. When they are both scaled from zero to one (rather than expressed as a percentage), multiplying by 2 ensures that F1 will also fall between 0 and 1, with larger being better. Often in a particular application, recall is more important than precision or the other way around. There is another formulation of the F-score that allows us to put a finger on the scale. Fρ = (1 + ρ2) × precision × recall /        ρ2 × precision + recall The factor ρ represents how much more important recall is than precision. For example, ρ = 2 indicates that recall is twice as important as precision, that is, that suffering two false positives is just as bad as a single false negative. (Fρ is usually presented as Fβ, but since we're already using β for upward-biased additive smoothing that would have been confusing.)
    💬 0🔄 0⭐ 0