In your bot detection systems I do not see open source system like anubis, safeline or sibuna (I am the author of sibuna). Sibuna is built as self-hosted alternative to cloudflare with console UI providing OWASP CRS, JA4 TLS signature from ingress, these are still necessary besides bot detection and cloudflare offer it as enterprise offering.
I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.
Based on trying to buy anything retail that is scarce, bot detection systems are definitely not working. I wonder how their system would do against https://refractbot.com/ or other popular retail bots.
To be fair, most bot detection can't and isn't trying to catch 100% of bots. Generally the idea is to prevent repeated abuse, rather than stop every single instance of it.
In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.
Did they use the browsers against sites they owned or 3rd party ones? If the latter, then lots of sites have Cloudflare Bot Management on, without actual strict restrictions on bots (a lot of sites using CLoudflare as a CDN for instance)
I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.
Also, the methodology just compare the true-negative detection rate for known negative, it seems?
> "Proof of Human identified 99% of automated sessions"
> "We measured detection rates for automated sessions but did not evaluate false-positive rates among human respondents."
Trivially easy measure to game if so. "isHuman(x) = return false" gets you a 100%.
In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.
If blocking 1 bot request is +1 then making a human solve a captcha should be something like -100. That's how we should grade.