Patch management can be a complex and time-consuming process, and because of this, patches to fix vulnerabilities may not be applied before a hacker is able to breach an organization's security. The majority of organizations are not aware of these vulnerabilities until they have experienced a breach, at which point it is frustrating to learn that deploying a simple patch could have prevented the breach altogether.
You cannot deploy a patch until it has been tested, and patches often require IT professionals to reconfigure and reboot systems that are difficult to take offline for long. A Ponemon survey showed that 88 percent of IT professionals are not solely responsible for patches but rather they usually have to coordinate with other teams, which can delay patching by an average of 12 days.
Another issue organizations run into is that once a new patch is announced, hackers begin working to exploit the vulnerability before it is patched. The Ponemon survey also indicated that it can take an average of "43 days to see a cyberattack once a patch is released for a critical... vulnerability".
OpenSSL Shared Library Vulnerabilities
OpenSSL patch management is ridden with challenges. The open-source software is a library of cryptographic applications that are intended to secure networks, and it is used on the majority of HTTPS websites.
Research has indicated that OpenSSL is the most targeted software in the world, accounting for approximately 19% of hostile activity globally.
The Heartbleed Bug is a major vulnerability in OpenSSL software. It attacks a memory handling bug in versions 1.0.1 to 1.0.1f of OpenSSL that allows hackers to decode 64 kilobytes of encrypted data in each heartbeat. "The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software."
Hackers can obtain access to the names and passwords of users, cookies, communications, content, and more by exploiting this dangerous vulnerability. Heartbleed can be a devastating attack because it can impact both the client and the server itself. In 2014, when Heartbleed came to light, an estimated 17 percent of the secure web services on the internet were vulnerable.
The Challenges of Identifying Vulnerable Libraries
One of the biggest challenges in identifying vulnerable servers is that they often end up in the application indirectly, instead of being directly used in the code. The applications themselves have dependencies on other applications, so they are not as easy to identify as applications that are being directly used.
These are a transitive dependency, where the library depends on code in other libraries to run properly. So, a developer may choose one specific library, but it is dependent on another library, which, in turn, is dependent on code in another library. The more complex the application, the more dependencies the library has. If one of these libraries is vulnerable, it can be harder to identify because it is not being used directly, so an admin may not know to check the others when looking for vulnerabilities in the system.
This is why many admins will simply reboot the entire server when they need to update a patch to handle a vulnerability. They may not know which services are using which libraries - making rebooting the entire server feel like the easier route to take. However, rebooting the system involves service degradations and outages. A server may take time to stabilize after a reboot, causing major disruptions to service. Sometimes, the servers will not come back properly once they have been rebooted.
Another problem with reboots is that they can leave a server vulnerable to attack. Even if the servers are patched manually, if the server is not rebooted, vulnerabilities may still exist in the shared libraries.
Identify Libraries that are Still Vulnerable to Attacks After Updates with an Open-Source Scanner
In order to assist the FOSS community and help administrators manage hundreds of servers using shared libraries, KernelCare released Uchecker - a scanner that examines Linux network servers and detects out-of-date libraries used by running processes both on disk and in memory. Uchecker was created during the development process of live patching service for shared libraries – KernelCare+. The scanner is free and open-source, distributed under the GNU General Public License.
The Uchecker (short from "userspace checker") works with all modern Linux distributions starting from the 6th versions.
You can scan your systems for outdated shared libraries by running a single command:
curl -s -L https://kernelcare.com/uchecker | python
In the scan results, administrators receive a list of outdated libraries with the following information:
- Process ID
- Process Name
- Shared library build ID
Figure 1: Uchecker Scan Results
Figure 2: Uchecker Activity Diagram
Visit Uchecker's Github page to learn more about the free and open-source scanner that identifies outdated shared libraries residing both on disk and in memory. Watch a demonstration of how Uchecker works in this video.
Administrators who patch libraries manually without a reboot run the risk of vulnerabilities persisting in-memory. Library patching often remediates vulnerabilities on disk, but outdated vulnerable libraries still remain in memory and can be exploited, even after being patched. For example, research suggests that OpenSSL is the most targeted software in the world, accounting for 19% of hostile activity globally, although patches for this vulnerability have been available for years. KernelCare’s Uchecker will find false negatives by correctly reporting vulnerable libraries running in memory that could be reported as updated by other scanners.
About the Author
Aleksandra Mitroshkina takes care of product marketing at KernelCare, CloudLinux Inc..