OPEN HONEST FOUNDATION
Structural correctness for software, and controlled measurement for the AI that writes it.
The Foundation is a Missouri nonprofit corporation. It governs two open software-quality standards, the credentialing that goes with them, and the registered research that tests whether the standards work.
THE SLOP AUDIT
SLOP AUDIT 1.0 · OCTOBER 2026 · (version and date proposed)
174 of 200 leading open source applications cannot be verified exhaustively by testing
Exhaustive testing means that every state the program can enter is known in advance and every one of them can be tested. The Foundation measured 200 of the most-starred public repositories on GitHub against that standard, 50 each in Python, Java, TypeScript, and C#. Each one carries at least 100 stars, is neither a fork nor archived, is an application rather than a library, and ships its own test suite. Those were the conditions of entry.
Of the 200, 26 came in under the threshold, and 25 of those are TypeScript front ends built on the functional-component pattern. That cluster is the one part of the corpus that already conforms. Set it aside, and 174 of the remaining 175 repositories are above the threshold. The exception is a small Java authentication library.
Exhaustive testability is one of the 18 dimensions the Slop Audit measures. The audit also scores reliability, maintainability, and security, and it maps every dimension to the published thresholds of 10 named compliance frameworks. The audit leads with testability because that result is the one nobody expected.
Check a repository
The hosted tool at try.slopaudit.org takes a public GitHub repository. The result below shows what a report looks like.
CAN BE VERIFIED EXHAUSTIVELY
openhonest/slop-audit · Python · 100 percent of program state is finitely testable.
VERIFIABILITY
4 / 4
RELIABILITY
5 / 6
MAINTAINABILITY
4 / 5
SECURITY
3 / 3
A rust mark means that dimension is a documented failure mode in code written by AI. A paler mark means the dimension is expected to be one and nobody has measured it. Hover either for the measurement. Hover the circled i for a plain-language definition.
| VERIFIABILITY | ||
| Mutable-state ratio | 0.00 | ISO/IEC 25010 |
| Decision-space coverage | 100% | ISO/IEC 25010 |
| Thread-safety surface | 0 sites | NIST SSDF |
| Type-escape density | 0.4% | ISO/IEC 25010 |
| RELIABILITY | ||
| Containerization | Yes | SOC 2 CC7.1 |
| CI/CD pipeline present | Yes | SOC 2 CC8.1 |
| Test determinism | Not run | ISO/IEC 25010 |
| Health-check endpoints | Yes | SOC 2 CC7.2 |
| MAINTAINABILITY | ||
| God-file share | 2.1% | ISO/IEC 25010 |
| Fuzzy duplication | 1.3% | ISO/IEC 25010 |
| Delete/add ratio | 0.41 | ISO/IEC 25010 |
| Pre-commit hooks | Yes | SOC 2 CC8.1 |
| SECURITY | ||
| Secret scan | 0 findings | SOC 2 CC6.1 |
| Dependency pinning | Yes | NIST SSDF |
| Trailing-whitespace density | 0.0% | ISO/IEC 25010 |
Four files were scoped out of this audit. See which, and why. How each number is measured at slopaudit.org.
What the full audit covers
LAYER 1 · L1 APPRENTICE
The audit computes 20 indicators from git history, static analysis, walks of the abstract syntax tree, and execution of the repository's own test suite.
LAYER 2 · L2 PRACTITIONER
The audit makes 18 inspections of the artifacts around the code, in deployment, infrastructure, and operations.
LAYER 3 · L3 LEAD
Judgment by a trained assessor, on the questions that no mechanical measure can answer.
LAYER 4 · L3 REVIEWER OR L4 ARCHITECT
Architectural synthesis, under development. The deliverables a SOC 2 audit asks for come out of what the first three layers already found.
A certified audit takes a certified person at every layer, and each level of the credential adds one layer. The Layer 1 analyzer is open source, so anyone can run it and read the numbers without certification.
Six of the 18 dimensions sit outside source code entirely, in deployment, infrastructure, and operations. Those are the layers that decide most of a running system's reliability.
LIMITS OF THE HOSTED TOOL
The hosted tool runs the static indicators of Layer 1 only. It never executes the repository's code, so it cannot report test determinism or find behavior that no test covers. It reports whether code can be verified, not whether a test suite does verify it. Layers 2, 3, and 4 run from the command line.
What the Foundation runs
Slop Audit
It measures 18 dimensions across four layers and scores them against the thresholds of 10 named compliance frameworks. Layers 1 through 3 are complete and Layer 4 is under development. Apache-2.0.
Honest Framework
A specification for writing code so that whole classes of defects cannot happen. The specification is language independent and designed to require no special training. Apache-2.0.
Certified Honest Practitioner
The credential that qualifies a person to run a certified audit. Four levels, and each one adds a layer.
Research program
Pre-registered studies on two questions. Whether an instrument measures what it claims, which covers the Foundation's own standards and the evaluation suites the AI field relies on. How competence emerges in a language model, which puts the field's assumption about scale to the test.
Research
The research ensures the integrity of the applied science. It tests the claims of the Slop Audit and the Honest Framework and subjects them to peer review.
It also tests the fundamental assumptions that the AI industry bases its decisions on. The Foundation's first peer-reviewed results show that a widely used AI evaluation for grammatical competence is, in at least some cases, measuring the algorithm that converts words into tokens. The earlier work behind that paper showed that scale is not the only thing that determines whether grammatical competence emerges.
Every study is deposited on the Open Science Framework before any data is collected, and the deposit states the result that would refute the claim. Institutional collaborators outside the Foundation run the validation work. Results publish open access with replication packages whether or not they support the standard.
REGISTERED · OSF DBSYG
Construct validity of the Slop Audit
Whether the instrument measures the properties it names. The deposit states the result that would refute it.
REGISTERED
The Honest Framework applied to code written by AI
Whether the specification lowers defect density in code written by AI.
REGISTERED
Example quality and AI code generation
Whether the quality of the reference code a model is given appears in the code it produces.
PUBLISHED
Grammatical competence from a six-year-old's share of language
A 125-million-parameter model trained on 92.5 million French words reached 86 percent on a native Quebec French grammar benchmark.
Independence
The charter and governance secure the integrity of the organization. Both are public, including the provisions that constrain the founder.
NOBODY PROFITS FROM THE ANSWER
Trustee compensation is limited to expense reimbursement and honoraria. There is no revenue share and no equity. Mark-derived revenue is restricted to uses the charter names.
ARTICLES NO VOTE CAN CHANGE
Six of the charter's articles cannot be amended by any procedure. Each one closes a route by which the Foundation could be captured or shut down. The Foundation is held to a short list of things it may do, and it is barred from directing technical work or setting commercial strategy. Trustees owe their duty to the mission rather than to any stakeholder, and they have to refuse amendments that would break these provisions. One trustee is independent of every commercial interest and holds a veto over licensing, branding, and certification. If the remaining trustees deadlock on a replacement, an outside institution picks one, and the Foundation can never have zero trustees. The trustees can take the marks back from a licensee that violates the mission. The Foundation cannot be dissolved while the mark is in use, and the trustees cannot hand the mark to a successor organization.
A SAFEGUARD OF LAST RESORT
The trademarks are held by a separate entity, the Open Honest Trust, rather than by the Foundation. If the Foundation ever abandons the purpose set out in its charter, the Trust can revoke the license to use the marks. The Foundation would continue to operate and would no longer be able to call its work Open Honest.
The founder is currently the sole trustee. The charter calls this the founding period and sets a deadline on it. Within five years of initial funding the founder has to seat the independent board: three trustees, one of whom can hold no commercial interest in the work and no employment with any licensee. The founder is barred from taking that seat. If the founder cannot act, a named legal successor has to seat the board instead.
Results so far
The Foundation was incorporated in Missouri on May 5, 2026.
SEPT 2026
PEER REVIEWED
A 125-million-parameter model trained on 92.47 million French words, roughly what a six-year-old has heard, scored 85.97 percent on a grammar benchmark written by native Quebec French speakers. The same work found that single-token zero-shot scoring, a standard way to evaluate models at this size, is dominated by artifacts of the tokenizer and the prompt template rather than by the model. Accepted at BabyLM 2026.
2026
PRE-REGISTERED
Two identical models were trained on matched French and English corpora, with one variable changed. The French model reached 100 percent on grammar probes at 197 million tokens and held it. The English model never got above chance through 4.3 billion tokens. Extended English training cut perplexity 42 percent over 219,000 further steps while grammar accuracy stayed exactly where it started. Perplexity is the field's usual measure of whether a model is learning, and here it moved while the thing it stands in for did not. Deposited at OSF SJ48B before data collection. Preprint.
2026
PREDICTION FAILED
In the same deposit the Foundation predicted that interleaving French with English would raise English grammar accuracy. It did not. English stayed at chance, and French fell from 100 percent to between 50 and 70 percent. The prediction was published as filed, with the result that refuted it. Preprint.
JAN 2025
PREPRINT
Across 864 experiments on eight models from five providers, giving a model explicit classification rules gained 15 to 24 points where it scored below 70 percent without them, and lost 5 to 20 points where it scored above. Every effect was significant at p below 0.05, and the threshold predicted the direction correctly on all six tasks. Data at OSF PCX2D.
MAR 2026
PREPRINT
One enterprise team was measured across 287 FTE-days under two process conditions, with the same people and the same tools throughout. Under a structured development process the codebase satisfied 18 of 18 enterprise dimensions. After process authority moved to non-technical stakeholders, with no tests, no quality pipelines, and no code review, it satisfied 2 of 18.
APR 2026
PRE-REGISTERED
A survey measured 200 public repositories, 50 each in Python, Java, TypeScript, and C#. Every one was drawn from the most-starred repositories in its language, carried at least 100 stars, was neither a fork nor archived, showed the marks of an application rather than a library, and shipped its own test suite. Across 2.2 million functions the survey counted how many functions depend on state they do not receive as a parameter. Where that share is high, exhaustive testing is not possible at any test budget, because each such function multiplies the space of behavior a test suite would have to cover. Median shares were 61 percent in Python, 53 in Java, 40 in C#, and 15 in TypeScript. Of the 200, 174 were above 15 percent, which means those systems are impossible to test completely. Setting aside the 25 TypeScript front ends that make up the conforming cluster, 174 of the remaining 175 are above it. Deposited at OSF DBSYG. Preprint. Run the same measurement on a repository.
APR 2026
PRE-REGISTERED
In the same survey, 26 of the 200 repositories came in under 15 percent. Of those 26, 25 are TypeScript front ends in the React ecosystem, and the 26th is a small Java authentication library. Over the last five years the React community moved to an architecture built on immutable state and one-way data flow, which is the pattern the Honest Framework specifies, and it did so for predictable rendering rather than for verifiability. Millions of programmers now write this way in production. Whether the frontend and backend split predicts the ratio is registered as a separate confirmatory study.
Take part
Programmers
Build a language implementation, or improve the Python reference implementation.
Organizations
Adopt the framework, audit existing code, or train practitioners.
Researchers
Replicate a study, propose an adversarial collaboration, or contribute a benchmark in your native language.
Governance contributors
Read the documents and propose amendments through the procedure each one defines.
Where things stand
BUILT
The Honest Framework specification is complete.
The Slop Audit standard is published.
The hosted Layer 1 tool is running.
Three studies are deposited on the Open Science Framework.
The governance documents are filed and public.
NOT YET
No study has reported results.
The Python reference implementation is nearly complete, and no other language implementation exists.
The practitioner curriculum is in development.
The mapping to ISO/IEC 5055 is on the roadmap.
The independent board has not been seated.