STANDARDS

Open standards and credentialing

The Slop Audit measures a codebase and reports what is wrong with it. The Honest Framework is a way to build code so that whole classes of those defects cannot happen. The Certified Honest Practitioner credential qualifies the assessors who run the part of the audit that needs a person. Implementations of the framework can be certified as conforming to it, once the specification has been validated.

Each standard works on its own. A team can audit a codebase without adopting the framework. A team can follow the framework without ordering an audit. Both standards are Apache-2.0.

The Slop Audit is a measurement instrument for software quality. It scores any codebase across 18 dimensions, against the published thresholds of named compliance frameworks. Run it on a public repository at try.slopaudit.org, or download it from GitHub and run it in your own environment.

The audit measures structure, not appearance

People usually use the word as an impression: code that looks rushed, careless, or machine-written. The audit replaces that impression with a score against named indicators and published thresholds. A polished codebase can score badly. Rough-looking code can score well.

The audit does not depend on the Honest Framework. The standard works on any codebase, in any language, whatever its architecture. Following the framework is one way to pass. Other architectures pass too.

The Foundation's implementation runs in Rust or Python and scores nine languages: C, C#, Go, Java, JavaScript, Python, Ruby, Rust, and TypeScript. C gets no coverage or determinism number, because C has no standard test runner or randomizer. The Foundation built that implementation as a proof of concept and released it for anyone to use. Anyone can write another implementation, in any language, against the same standard.

The audit runs in four layers across 18 dimensions

Every layer of a certified audit takes a certified person. Each level adds exactly one layer, so an L3 Lead can run Layers 1 through 3. That is Phase 0 of the credential. Layer 4 is Phase 1, and it takes an apprenticeship rather than a course.

The Layer 1 analyzer is open source. Anyone can download it and run it on a repository, and the hosted version runs it in a browser. A result from either one is a measurement, not a certified audit.

The 18 dimensions cover security architecture, data architecture, compliance engineering, operational security, performance engineering, operations, DevOps, infrastructure, software architecture, governance, process engineering, lifecycle management, and software development. Six of them sit outside source code entirely, in deployment, infrastructure, and operations.

The frameworks it maps to are SOC 2 Trust Services Criteria, NIST SP 800-53, OSFI B-13, OWASP ASVS, ISO/IEC 25010, WCAG 2.2, Section 508, EN 301 549, AODA, and Quebec Law 25.

The audit measures properties directly wherever it can

Most software metrics are substitutes. Cyclomatic complexity, which nearly every code-quality tool reports, counts the decision branches in a program and treats that count as a stand-in for a property static analysis cannot measure. Anyone who uses a stand-in has to show that it tracks the property. Nobody has shown that for cyclomatic complexity. Researchers have reported a stable linear relationship with lines of code, which suggests it mostly measures size.

The audit counts the property itself wherever a direct count is possible. To measure how much of a program's state is mutable, it builds the abstract syntax tree and counts how many lines write to each memory location. To measure whether a test suite gives the same answer twice, it runs the suite more than once and compares. To find behavior that no test covers, it runs the tests and checks what they never touch. Most of the 20 Layer 1 indicators work this way.

Nobody has shown that a high mutable-state ratio leads to more defects in production. The research program's registered validity studies are designed to test that.

The first measurement covered 200 repositories and one indicator

The Foundation's first registered audit experiment measured the mutable-state ratio across 200 of the most-starred public repositories on GitHub, 50 each in Python, Java, TypeScript, and C#. Each one carries at least 100 stars, is neither a fork nor archived, is an application rather than a library, and ships its own test suite.

Across 2.2 million functions, median shares of functions that depend on state they do not receive as a parameter were 61 percent in Python, 53 in Java, 40 in C#, and 15 in TypeScript. Of the 200 repositories, 174 were above 15 percent, which means those systems are impossible to test completely. Setting aside the 25 TypeScript front ends that make up the conforming cluster, 174 of the remaining 175 are above it.

The analyzer went through three iterations. The first two produced systematically wrong numbers that passed the author's own checks, and both result sets are preserved in the replication package alongside the third. The third was validated against a 249-scenario behavioral specification before its results were accepted.

The Honest Framework is a specification for writing code that is correct by construction. It is language independent, and it is designed to require no special training. The audit reports how much mutable state a codebase carries. The framework is how a team makes that state finite and deterministic.

What the specification requires

These requirements are unrelated to each other. A codebase can satisfy one without the other. Each one shuts down a different way for a program's behavior to become impossible to test.

Together they produce code that a person can check by reading it, instead of code that someone can only check by running it and checking every possible path.

Nothing about the specification requires a new language

Most tools for provably correct software require a different language, such as Haskell, Idris, or Coq, or a fight with a demanding type system that requires extensive retraining. Honest works in any object-oriented or imperative language, using the syntax a team already has.

A programmer can write honest code without the framework, and many already do. The framework supplies a toolchain that constrains less experienced programmers and AI agents so that they have to write this way.

Millions of programmers already write this way

Correct-by-construction methods have a long history of staying in academia. The Honest Framework is based on Honest Code, a book by the Foundation's founder written to make those methods usable by the average programmer. There is little new to learn. The book asks for a few small changes in habit, and most of them mean dropping a practice rather than picking up a new skill.

The pattern removes whole categories of defect instead of catching them in testing.

The React community arrived at most of the same pattern on its own. Function components and hooks shipped in February 2019, and over the years that followed the community moved to immutable state and one-way data flow. The pattern has run at production scale ever since. In the Foundation's survey of 200 public repositories, 26 came in under the mutable-state threshold. Of those 26, 25 are TypeScript front ends in the React ecosystem, and the 26th is a small Java authentication library. React programmers adopted the pattern for predictable rendering and easier debugging, not for verifiability.

The Rust compiler enforces most of the specification's memory requirements. The borrow checker allows only one owner of a mutable value at a time, so writes to that value cannot come from two places at once. The compiler refuses to build a program that breaks the rule, so a Rust programmer satisfies it without reading any specification. Rust has grown steadily since version 1.0 shipped in 2015. Rust repositories almost always pass the mutable-state indicator in informal checks. This is why Rust was not included in the survey.

React only covers one layer of one kind of application, and using Rust requires retraining. The Honest Framework applies the same pattern in any language that supports first-class functions, which includes Python, Ruby, Java, JavaScript, Go, and C#. The framework a team already runs keeps running. Rails, Django, Spring, Express, and ASP.NET all stay in place.

Updating existing code written in established frameworks happens one function at a time. A programmer takes the business logic out of one method, puts it in a pure function, and has the method call the function. The class is one method lighter and its interface has not changed. The next sprint takes another method. This is the strangler pattern. There is no rewrite and no migration plan. After enough sprints the old framework handles routing and I/O, and the business logic has moved into pure functions.

A machine can follow the specification without interpreting it

Conventional coding standards are written for a person to read and judge. Their value depends on a skilled reader who supplies the taste to apply them. Two programmers given the same standard will build two different systems.

A machine supplies no taste. It follows what it is given, so a standard that depends on human judgment loses its value when a machine applies it.

Neither requirement depends on judgment. A decision table either is a table or is not. A memory location either has one write site or has more. A checker can verify both.

Where it applies

Anywhere code has to be correct. Scientific code including climate models, physics simulations, and AI systems. Financial systems. Healthcare records and decision-support tools. Civic infrastructure. Security tools. Voting and election systems. Accessibility software. Any other safety-critical or compliance-sensitive system.

A certified audit takes a certified person at every layer. The credential runs in four levels, and each level adds one layer. An L1 Apprentice runs Layer 1. An L2 Practitioner runs Layers 1 and 2. An L3 Lead runs Layers 1 through 3, which is the whole of Phase 0. Layer 4 belongs to an L3 Reviewer or an L4 Architect, and that level comes from an apprenticeship rather than a course.

Certification is the only part of either standard where the Foundation licenses a mark. One mark goes to a person who administers the Slop Audit. The other goes to an implementation of the Honest Framework. Certification is the only source of mark revenue, and the charter limits where that revenue can be used. See Governance.

An implementation of the Honest Framework can be certified as conforming to it. The conformance suite of roughly fifty laws is the test, and an implementation either passes it or does not. A certified implementation can use the Certified Honest Framework mark.

Certification will be free for implementations released under a free and open source license. A proprietary implementation will pay a royalty for use of the mark. Both pass the same conformance suite.

The Foundation will administer that certification after the specification itself has been validated. Studies B, D, and H are that validation. They are registered on the Open Science Framework and awaiting funding. See Correctness research.

ISO/IEC 5055

ISO/IEC 5055, published in 2021, counts occurrences of specified critical weaknesses in source code across four characteristics: security, reliability, performance efficiency, and maintainability. The counting is mechanical and uses static analysis. The security and reliability measures take their weaknesses from the Common Weakness Enumeration, a catalog built from documented incidents, so both have evidence behind them.

ISO/IEC 5055 defines reliability as the degree to which a system performs specified functions under specified conditions for a specified period of time. That definition describes a running system. Redundancy, health checks, capacity, and deployment practice decide most of it. The measure reads source code and never runs the system or its tests. Nobody has shown that the count predicts the behavior in the definition.

Some of what ISO/IEC 5055 counts is a matter of style. The maintainability measure counts cyclomatic complexity, component size, and duplication, which record what a committee agreed good code looks like rather than how a system performs. The Slop Audit scores measured properties of a codebase and its behavior. No indicator in it scores style. A program built entirely from objects, with no pure functions at all, can pass the audit if the programmer was rigorous enough.

The Slop Audit's mapping to ISO/IEC 5055 is on the roadmap. It will cover security, reliability, and performance efficiency. It will not cover maintainability.

The Slop Audit also runs the repository's own test suite. That is how it measures test determinism and uncovered behavior, which static analysis cannot find on its own.

ISO/IEC 42001

ISO/IEC 42001 is the management-system standard for organizations that build or use AI. It asks an organization to show how it governs AI work, not what the resulting code looks like. The Slop Audit measures the code. The two standards answer different questions. An organization that is pursuing 42001 certification can use the audit as evidence for the parts that concern software quality.

The Foundation's position on ISO/IEC 42001