mspm: A revive Pass for a Small Go Pattern Matching Library

mspm is a small Go library for matching a set of literal byte string patterns against a document and counting the hits. This week it received exactly one commit, Initialize Go module and refresh README, but that commit is worth a look. It adds a go.mod and replaces an outdated README with one that documents real behavior, real limitations, and a working example. The project explicitly calls itself “being revived”.

The change touches two files: README.md and go.mod. 63 insertions, 33 deletions. No library code changed.

The go.mod is new. It declares module github.com/BlackRabbitt/mspm and requires go 1.27.0. Before this commit the module path existed only implicitly, which made go get behavior unreliable for anyone importing the search package directly. Adding the module file is a small fix with practical value: it makes the library installable with a plain go get github.com/BlackRabbitt/mspm and pins the language version CI will build against.

The old README showed a usage snippet that referred to package names which no longer match the tree. The new one is a working, copyable example using the current search package:

model := search.NewModel("keywords")
model.Build(strings.NewReader("mspm\npattern\nsearch\n"))

counts, err := model.MultiTermMatch(strings.NewReader("mspm pattern search mspm"))
fmt.Println(counts["mspm"]) // 2

Patterns go into Build one per line. Matching is case-sensitive. MultiTermMatch returns a map of counts keyed by the matched pattern. That is the entire API surface, and the README now says so plainly instead of pointing at stale godoc badges.

The most useful part of the rewrite is the “Current behavior and limitations” section. The project lists four constraints that any potential user needs to know before wiring this into a production pipeline:

  • It works on bytes, not Unicode aware characters. Multibyte runes can match in ways that surprise you.
  • Overlapping matches are not all reported. The scanner consumes matches as it goes.
  • The matcher is trie based but does not implement Aho-Corasick failure links. Text that partially matches and then diverges forces a restart of the scan, so worst case throughput degrades.
  • Build does not return scanner errors from its input reader. An unusually long pattern line can be truncated or dropped without the caller seeing an error.

The failure link point matters most. A full Aho-Corasick automaton guarantees linear scan time in the document length regardless of how many patterns fail to match. Without it, the search falls back toward repeated prefix rescans. For a short keyword counting job this is fine. For scanning large log volumes against thousands of patterns it likely is not, and the README is honest about that.

The Build error swallowing is the classic footgun. If a pattern line exceeds the internal bufio scanner buffer, the caller has no way to detect it. Counting against a silently incomplete pattern set produces wrong results with no signal. Anyone using this today should either keep pattern lines short and verified or preflight the input.

Prior activity included benchmarks across a trie package comparing plain trie nodes against hash backed nodes, with search costs around 200 ns/op on amd64. Those benchmarks are referenced by the old README which is now gone. Restoring them, updated for the current tree, would make the performance claims testable again.

  • If you already depend on mspm, add the module path in your own imports as github.com/BlackRabbitt/mspm and re run go mod tidy after pulling the new go.mod.
  • Pin your expectations: byte semantics, no overlap reporting, no failure links. If any of those three matter to your workload, use a full Aho-Corasick implementation instead for now.
  • The README names go test ./... as the test entry point. Run it before upgrading since there is no tagged release or changelog to rely on.

The project lives at github.com/BlackRabbitt/mspm on branch master. One commit is a small window, but the commit that exists moves the library in the right direction: documentation that tells the truth and a module file that makes it installable. Watch for the failure link implementation and an error return from Build as the next signals that the revival is real.