Breaking Changes, Breaking Trust
Breaking Changes, Breaking Trust
Breaking Changes, Breaking Trust
Breaking Changes, Breaking Trust
Breaking Changes, Breaking Trust
Breaking Changes, Breaking Trust
Every developer has experienced it. They’re working smoothly on a project when they realize they need to upgrade React from version 17 to 18 due to a security vulnerability or maintenance requirement. It seems straightforward enough. They make the update, run their tests, and everything appears fine. But weeks later, they start noticing incompatibilities. Components that once rendered flawlessly are now throwing unexpected errors. They’ve just encountered a hidden breaking change. When it comes to relationships between development and security teams: Breaking changes break trust.
This scenario is common in software development. What initially seemed like a routine update has cascaded into a series of unforeseen issues. But to make matters worse, we toss in an SLA for the fix. Security fixes are often critical, but they can introduce breaking changes that disrupt the flow of development. This puts developers in a tough spot—balance the immediate need to patch vulnerabilities against the risk of destabilizing their applications. Security teams often focus on fast vulnerability fixes, while developers aim to maintain stability and avoid disruption. When a security patch introduces a breaking change, the developer’s response might be to delay the update or try to work around it, which can lead to vulnerable systems staying exposed longer than necessary.
To run an effective security program, practitioners should understand the concept of breaking changes and why they occur. Let's begin by understanding what constitutes a breaking change and why they're both challenging and sometimes necessary in the software development lifecycle.
What is a breaking change?
Breaking changes in software are modifications that disrupt compatibility with previous versions of libraries, frameworks, or systems. There are broadly two categories of changes that break backward compatibility: syntactic changes and behavioral changes.
Syntactic Changes
Syntactic changes are modifications that break existing contracts or interfaces between different parts of a software system. These changes alter the code structure, invalidating previously established agreements on how components interact. When such a contract changes, it necessitates developer intervention to update the code and reestablish a valid contract. This requirement for code modification classifies it as a breaking change.
Syntactic changes can often be detected by compilers. For instance, altering a method's signature, removing a class, or modifying an API endpoint will trigger immediate compilation errors. This instant feedback makes syntactic changes straightforward to identify, though potentially challenging to resolve in complex systems.
Consider these code examples illustrating how syntactic changes affect contracts between components:
Code example with original contract
public class UserService {
public User getUser(int id) {
// Implementation
}
}
// Usage
UserService service = new UserService();
User user = service.getUser(123);
Here, the contract stipulates that getUser accepts an integer id and returns a User object, as demonstrated in the usage example.
Code example with syntactic breaking change
public class UserService {
public User getUser(String id) {
// Implementation
}
}
// Usage
UserService service = new UserService();
User user = service.getUser(123); // Compiler error: incompatible types
The contract now requires a String parameter, breaking the existing agreement between UserService and its clients.
To resolve this, developers must update all getUser calls:
User user = service.getUser("123");
This modification reestablishes a valid contract but may require extensive codebase changes, affecting multiple components relying on UserService. While seemingly simple, such changes can have far-reaching, costly implications for organizations. The Python 2 to 3 transition exemplifies this: what appeared to be minor syntactic changes, like modifying print "Hello" to print("Hello"), evolved into a decade-long, industry-wide transformation, underscoring the profound impact of syntactic changes on software ecosystems.
Behavioral changes
Behavioral changes modify a system's functionality without altering its interface structure. These changes affect operation outcomes or side effects while preserving method signatures and API structures. This characteristic makes behavioral changes difficult to detect: existing code typically continues to compile and run without immediate errors, obscuring potential problems.
The main challenge with behavioral changes is their potential to break assumptions about method behavior, causing unexpected results in dependent systems. Issues often only emerge during runtime or in specific use cases, complicating debugging. This combination of delayed problem detection and silent failures makes behavioral changes particularly challenging in software development. Mitigating these risks requires comprehensive testing strategies, clear documentation of changes, and effective communication within development teams.
Consider these code examples illustrating how behavioral changes affect the functionality of components:
Code example with original behavior
public class MathUtils {
public static int round(double value) {
return (int) Math.floor(value + 0.5);
}
}
// Usage
int result = MathUtils.round(2.5); // result is 3
int result2 = MathUtils.round(-2.5); // result is -2
Here, the round method uses the common rounding rule of adding 0.5 and then flooring the result.
Code example with behavioral breaking change
public class MathUtils {
public static int round(double value) {
return (int) Math.round(value);
}
}
// Usage
int result = MathUtils.round(2.5); // result is 3
int result2 = MathUtils.round(-2.5); // result is -3
The method signature remains the same, but the behavior has changed to use Java's Math.round(), which rounds differently for negative numbers. This modification alters the method's behavior without changing its signature. Existing code will still compile and run, but the results will be different for negative numbers
A real-world example of such a behavioral change is the modification of Python's round() function between versions 2 and 3. In Python 2, round(0.5) would return 1, while in Python 3, it returns 0 due to the adoption of the "round half to even" rule. This change, while improving statistical accuracy, could lead to unexpected results in code migrated from Python 2 to 3, illustrating the subtle yet significant impact of behavioral changes.
SemVer was introduced to communicate software changes effectively
Syntactic and behavioral breaking changes create challenges for library maintainers, so they need a clear way to communicate updates. Semantic Versioning (SemVer) was developed to address this.
SemVer provides a systematic approach for signaling the type and impact of changes in software releases. It uses a three-part version number (MAJOR.MINOR.PATCH), with each part indicating different levels of change:
- MAJOR: Incremented for incompatible API changes
- MINOR: Incremented for backwards-compatible feature additions
- PATCH: Incremented for backwards-compatible bug fixes
This system allows maintainers to effectively communicate whether a new version contains backwards-compatible changes or breaking changes, enabling dependent projects to update with confidence or prepare for potential disruptions.
SemVer falls short in communicating software changes clearly
SemVer's effectiveness hinges on maintainer judgment. The subjective nature of SemVer leads to inconsistencies across projects and even within single projects over time. A study by Ochoa et al. revealed that 21.9% of breaking changes in Maven Central were not documented, highlighting the potential for unexpected issues in seemingly safe updates. This subjectivity creates a hidden "tax" on estimating upgrade efforts and undermines developers' ability to confidently adopt new library versions.
Moreover, while SemVer is designed to signal changes, it struggles to adequately convey behavioral changes. It often fails to represent subtle yet impactful modifications, particularly those that don't alter the API structure but affect the underlying functionality. This limitation is especially problematic for complex systems where small changes can have far-reaching consequences.
The combination of SemVer's subjective interpretation, its limitations in capturing behavioral changes, and the inadequacies of current automated tools creates an environment of uncertainty. Developers must navigate this landscape carefully, balancing the need for up-to-date dependencies with the risks of introducing unforeseen issues. This delicate balance often leads to cautious upgrade practices, potentially resulting in delayed adoption of important security patches or performance improvements.
So if we can’t rely on SemVer or documentation what can we trust? We can trust empathy as a security strategy!
How to build developer empathy into security practices
The friction between development and security teams often stems from competing priorities: developers want stable, uninterrupted workflows, while security teams push for timely fixes to vulnerabilities. This can create tension, especially when updates introduce breaking changes that disrupt existing systems. Yet, empathy can bridge this gap—transforming the relationship from a push-pull dynamic into a collaborative one.
For developers, the burden of dependency updates includes regression testing, fixing bugs, and ensuring that changes don’t cascade into ecosystem failures. Recognizing this, security teams can foster a smoother update process by aligning on two critical fronts: visibility and communication.
Empathy-driven risk communication
Security teams can make it easier for developers to prioritize updates by providing context on risk and urgency. Describing vulnerabilities with clear risk assessments—like "this is low-risk but recommended" versus "high-risk and potentially exploitable in production"—allows developers to better balance their workload.
Enhanced transparency on breaking changes
Tools that provide detailed visibility into which changes and vulnerabilities will affect the application—like Endor Labs’ reachability analysis—help reduce unnecessary tension and ensure smoother updates, aligning security needs with development workflows
Flexible SLA models for complex updates
Security teams should acknowledge when a breaking change is going to be difficult to implement. Flexibility in SLAs such as usage of Digital Ocean’s security debt approach allow for complex updates to still be tracked without the pressure and friction of a strict SLA. Reporting on debt can bake in reasonable paths for report, especially when some SLAs may prove to be extremely disruptive. These approaches can go a long way in building trust. This allows developers to feel more supported when handling challenging updates without the added pressure of a rigid deadline.
Empathy helps bring these two perspectives together. It’s not just about minimizing vulnerabilities; it’s about ensuring the application remains resilient and stable in the process. By taking steps to align security priorities with the realities of development workflows, we can create a more integrated approach to software security. After all, availability is as crucial to the CIA triad as confidentiality and integrity.
A closing call to action
For security and development teams alike, a foundation of empathy improves both collaboration and outcomes. If you’re interested in tools that help you see the impact of changes more clearly, explore Endor Labs’ reachability analysis and automated update assessments. In an upcoming post, we’ll dive deeper into Endor Labs’ upgrades and remediation capabilities—tools designed to help you strike that balance between security and stability.
References
[1] Ochoa, L., Degueule, T., Falleri, J. R., & Vinju, J. (2021). Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central. arXiv preprint arXiv:2110.07889.