Maybe I missed it, but it look like this has just a single metric. Maybe instead of making a new project, you could try to get this metric added to a existing tool like https://dekobon.github.io/big-code-analysis/index.html which already has dozens of metrics.
Yes, I'm doing my own research on AI augmented pipelines https://blog.officefloor.net . I actually found most code quality tools look for bugs and complexity, but nothing much about cohesive erosion. The nice thing about this metric, is that it determine the files where the erosion is occurring. I turned it into a GitHub action to make it easier to access to get wider feedback on the metric. The GitHub action triggers on your merge request and tells you the files where erosion is occurring to refactor. This stops erosion before it gets too expensive to change (big refactors or rewrite). Yes, happy to work with others to get the metric into other tools.
I've never encountered that term before (cohesive erosion) but I like it, if I'm interpreting it correctly.
Do you mean like the hyper focus an LLM puts on the task in front of it so you end up with drift (duplicated concepts/multiple ways of doing things, terminology drift (e.g., now we have "customer" and "client"). That sort of thing?
When you think about a god class or god method, it occurs over time by adding more than a single responsibility.
Yes, there are generally complex algorithms but they usually are not things developers write (imported from libraries).
What is usually going on in the god class/method is that things keep getting added to it. These things should be separated out. So the cohesiveness of the class/method erodes into doing too many things.
The idea of the Change Impact formula is to catch this early so you start refactoring to separate out into classes with single cohesive purposes.
The problem with AI is it handles complexity really well and will happily keep piling changes into god classes/methods reaching ridiculous CC levels (have see over 200). Previously developers would get annoyed and do the refactor. But with AI these days, changes are happening faster. So Change Impact is to try to monitor the cohesive erosion.
That project in itself looks very interesting. How are people using it, any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Code metrics in general aren’t that widely used. I’ve only ever worked at one place (a bank) that tracked it, and that was only because sonarcube had it built in.
While a lot of metrics make intuitive sense, we don’t have that much hard evidence to prove or disprove their value. Part of it is the whole “if a metric becomes a target, it ceases to be a good metric” thing. Adding the checks to a large existing project probably has negative value. But I think it’s worth doing for greenfield projects.
For humans, these should just be advisory. But for LLMs I’m happy enough to make it a blocking check.
I keep thinking of doing an experiment where I give the same LLM the same problem, and only change which metric is enforced. And then see if any of them have a noticeable effect on correctness/maintainability.
> any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Yeah they have examples of adding it to CI, or local checks, generate html reports, etc in their docs.
Nice idea. Our new CTO brought in a tool he made for analyzing cyclomatic complexity and it’s been useful since we’re a heavily AI-forward shop.
BTW, you can avoid your comments being flagged and killed by writing them yourself! I know it’s tempting to offshore it to AI (especially after you’ve vibe-coded a whole project) but some genuine human communication goes a long way.
I know cyclomatic complexity has been heavily debated for a long time, but I do think it’s valuable. It’s really good at highlighting common annoyances like overly clever code, nested ternaries, dense functions with too many branches…
The only thing is that these issues seem like human code problems and IME LLMs don’t really write code like this anymore. It’s almost the opposite in python, actually, where Claude leans on writing lots of 2-3 liner private utils which is a separate kind of complexity and organization problem.
I still find it useful specifically for React where it’s frustratingly normalized to write many branches in your JSX though.
The difference to previous CC use, is the the change impact formula looks at the complexity already in the class/file. Typical CC just looks at the function it is change and not the context of the change. The Change Impact formula incorporates that to avoid god class and god method issues. Plus multiplying by number of files punishes for non-cohesive code bases. For me it puts the intuition of high cohesion and low coupling into a measurable metric.
Personally I’m ok with vibe coded projects - certainly feels like the future of things, and I think the line between vibe coded and “professionally” coded is increasingly blurring - but I completely agree on the marketing/communications piece. Ideally your communication about a project conveys real expertise and ownership, signaling that you really understand the problem you’re trying to solve and the tradeoffs you made in your approach to do so. I am very hesitant to use a project where it feels like the eng couldn’t pass a pop quiz about how it works + why.
Yes, I've run it against a bunch of open source projects with long histories (before AI) to see if it predicts bugs. Seems file size is still a better predictor. However, for the projects where good coding was strictly adhered to and others that were not, it showed the differences appropriately. So I've found it useful in general for Software erosion.
Do you mean like the hyper focus an LLM puts on the task in front of it so you end up with drift (duplicated concepts/multiple ways of doing things, terminology drift (e.g., now we have "customer" and "client"). That sort of thing?
Yes, there are generally complex algorithms but they usually are not things developers write (imported from libraries).
What is usually going on in the god class/method is that things keep getting added to it. These things should be separated out. So the cohesiveness of the class/method erodes into doing too many things.
The idea of the Change Impact formula is to catch this early so you start refactoring to separate out into classes with single cohesive purposes.
The problem with AI is it handles complexity really well and will happily keep piling changes into god classes/methods reaching ridiculous CC levels (have see over 200). Previously developers would get annoyed and do the refactor. But with AI these days, changes are happening faster. So Change Impact is to try to monitor the cohesive erosion.
While a lot of metrics make intuitive sense, we don’t have that much hard evidence to prove or disprove their value. Part of it is the whole “if a metric becomes a target, it ceases to be a good metric” thing. Adding the checks to a large existing project probably has negative value. But I think it’s worth doing for greenfield projects.
For humans, these should just be advisory. But for LLMs I’m happy enough to make it a blocking check.
I keep thinking of doing an experiment where I give the same LLM the same problem, and only change which metric is enforced. And then see if any of them have a noticeable effect on correctness/maintainability.
> any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Yeah they have examples of adding it to CI, or local checks, generate html reports, etc in their docs.
BTW, you can avoid your comments being flagged and killed by writing them yourself! I know it’s tempting to offshore it to AI (especially after you’ve vibe-coded a whole project) but some genuine human communication goes a long way.
The only thing is that these issues seem like human code problems and IME LLMs don’t really write code like this anymore. It’s almost the opposite in python, actually, where Claude leans on writing lots of 2-3 liner private utils which is a separate kind of complexity and organization problem.
I still find it useful specifically for React where it’s frustratingly normalized to write many branches in your JSX though.