Farley Score/Index

The Farley Score (or Farley Index) is a 0 to 10 metric used to evaluate the quality of software test suites based on continuous delivery pioneer Dave Farley's core properties of good tests.

The 8 Properties of Good Tests

Tests are assessed against key design dimensions:

• Understandable: Easy to read and grasp intent quickly.
• Maintainable: Resilient to code changes.
• Repeatable: Deterministic and produces consistent results.
• Atomic: Runs independently of other tests.
• Necessary: Provides real value and guards against defects.
• Granular: Tests a single specific behavior or unit.
• Fast: Executes quickly to support rapid feedback.
• First: Written using test-driven design (TDD) principles.

Rating Scale

• 9.0 – 10.0: Exemplary
• 7.5 – 8.9: Excellent
• 6.0 – 7.4: Good
• 4.5 – 5.9: Fair
• 3.0 – 4.4: Poor
• 0.0 – 2.9: Critical

source: Farley Score Plugin

source: TDD - The Properties of Good Tests by Dave Farley

My thots:

Maintainable

"Resilient to code changes" this AI summary leaves a bit to be desired. The "TDD" article linked above reads,

Maintainable - We’d like our tests to act as a defence of our system, breaking when we want them to, but also remaining true as we change our system. So they need to be maintainable so that they are easy to change, without losing their intent.

Ideally, any behavioral code change will break a test. A code change without a test change is a warning sign. That said, no code, not even test code, should be fragile in the sense that changes introduce defects.

Repeatable 

"Deterministic and produces consistent results." Agreed. I have seen some tests make extensive use of random data generators. And there are benefits, but deterministic is best where possible (e.g., date/time may be difficult). Random data may help find defective edge cases, but if not deterministic, recreating the error may be challenging.

Absent: Code Coverage 

From this statement, "Dave Farley explores why test coverage is a poor metric for success, how one company replaced manual testing with unit tests that gave high coverage and no protection and paid people extra to do it.", ignoring code coverage may be intentional.

I'm a strong advocate for 100% code coverage, allowing for very limited labeled exclusion (C# e.g., [ExcludeFromCodeCoverage]), for both production code and Unit Tests. Personally, I have discovered many faulty unit tests through missing code coverage of the unit test code.

Some have said, code coverage doesn't tell you the code is good. And, it may be covered but missed important cases. I agree, 100% is the minimum metric for tests. Anything less means code is not tested and may break and is not protected against unintentional behavioral changes. Many error log messages causes errors themselves due to defective arguments that were never tested. And errors are often most difficult to test manually. And manual testing offers no protection for the future. It seems many people interpret "test coverage is a poor metric for success" to mean 80% is good enough. It means test coverage is a poor metric for success because it pins at 100%, which is the minimum. Test coverage is a metric of risk exposure, especially to future change, especially with coders unfamiliar with the code.

TDD

I think TDD is fine, but I'm not a strong proponent. For fixing bugs found in production, YES, TDD is the way to go. Ensure the problem can be recreated before it's fixed.