Web Security 101: Proxies with HTTPS Inspection, Working Together
By Douglas Foster
Web proxies with HTTPS inspection provide important layers for any network defense-in-depth strategy. They provide a first-stage defense against inappropriate usage and malicious web content, a second-stage defense against email containing hostile links, and a tertiary defense against infected machines seeking a command-and-control server. A simple proxy implementation will inspect and protect unencrypted traffic, while HTTPS inspection evaluates encrypted traffic.
Web threats come from several directions: typing errors and poor judgments that send users to dangerous sites; legitimate sites that have been infected with malicious content or links; malicious ads that sneak onto legitimate sites; malicious links launched from email or other socially engineered attacks; and, finally, DNS or traffic-routing attacks that redirect users away from their intended server. It is unreasonable to assume that users can defend themselves and the internal network from this collection of threats.
Web Proxy Configurations
A Web proxy’s primary benefit is blocking web requests to sites that might be dangerous. This decision process depends on access to a database that rates websites for risk (malware threats) and category (compliance with acceptable use policy). This information is the most valuable aspect of the product purchase, and its subscription fees are likely to be the largest component of the proxy device’s total cost of ownership (TCO).
For the proxy to work, web traffic must flow to or through the device. Standard proxies are configured at the client, and explicitly direct the web browser to forward web requests to the proxy device. The proxy device then performs the DNS lookups to convert names to numbers, as well as evaluating the request for acceptability. Because the browser and proxy communicate directly, the level of protection and the quality of log information will typically be superior to using a standard proxy.
Since some devices and applications will not comply with the proxy configuration instructions, and since malware may deliberately attempt to bypass the proxy, it is also desirable to configure the proxy on the path to the internet, so that it can inspect all traffic for web packets. This mode of operation is called a transparent proxy. Standard proxy packets are addressed to the proxy device, usually at port 3128 or 8080. Transparent proxies are addressed to an internet address, using port 80 or 443, but pass through the proxy due to network configuration. New administrators may feel compelled to choose between standard or transparent mode, but the most secure approach is to implement both inspection methods.
HTTPS-encrypted traffic is a growing portion of all internet traffic. Web researchers currently estimate that it exceeds 50 percent of all internet traffic, and my research suggests a much higher percentage for typical business usage of the internet. LetsEncrypt.org provides a free and automated system for certificate issuance, which is expected to accelerate this trend. As a security measure, HTTPS encryption is highly desirable because the certificate verification process ensures traffic has not been misdirected by DNS or network routing attacks. HTTPS encryption also ensures that packets cannot be maliciously modified in transit.
HTTPS encryption is not a single technology; rather, it’s a set of layered modular components that achieve the desired result. White hat researchers have been carefully inspecting each of these modules, and many of the older component technologies have been criticized as a result of this research. Consequently, older devices may not be able to generate an encryption session that meets the regulatory and security requirements of the organization. Similarly, web browsers implement different responses to invalid certificate chains, often leaving the final decision in the hands of the end user. A tandem web proxy provides the organization with the ability to enforce a typical encryption standard for traffic entering the internet, enforce a standard response to invalid certificate chains and establish a standard process for implementing exceptions.
However, HTTPS encryption is also intended to prevent traffic from being understood or altered by man-in-the-middle devices, such as a web proxy. If a web proxy is excluded from the HTTPS session, it cannot enforce encryption standards, cannot enforce certificate standards, cannot block hostile web content and cannot usefully log web activity. Because these are important security considerations, a bit of deliberate deception is needed to allow the web proxy to inspect HTTPS traffic.
This is how the “useful deception” works:
The proxy is configured as a Certificate Authority.
The proxy CA root certificate is distributed to all internal devices. This ensures that these devices will accept any certificate issued by the proxy.
When a client system attempts to connect to, say, www.example.com, the proxy connects to the web server on the user’s behalf, and then the proxy responds to the user with a certificate for www.example.com issued by the proxy CA. Because there is now one session from the browser to the proxy, and a second session from the proxy to the web server, the proxy is able to decrypt, inspect and re-encrypt the HTTPS traffic. The external HTTPS session is encrypted based on the capabilities and configuration of the proxy, while the internal HTTPS session can be implemented with weaker encryption standards if needed to accommodate legacy devices.
HTTPS inspection is somewhat controversial, largely because of research that has exposed weaknesses in different vendors’ implementations. These include weaknesses in the encryption configuration (often accompanied by poor visibility into the configuration); failure to block sites with invalid certificates; or unnecessary blocking of sites with correctable certificate errors. These objections are legitimate, but they are not so much a reason to reject HTTPS inspection as a reason to be an intelligent consumer. Just as standard and transparent proxy modes should be used together, HTTP and HTTPS inspection should also be used together. Since a chain is only as strong as the weakest link, organizations cannot afford to allow weak links to persist.
Authentication and Logging Issues
To provide accountability and policy enforcement, a web proxy should identify the user associated with each web request. For the Windows environment, this can usually occur transparently, using NTLM (NT LAN manager) information that modern browsers include with each web request. Alternate login methods should be available for situations where identity cannot, or should not, be determined automatically. Authentication permits the proxy to enforce allow/block rules based on the user and group, as well as linking a user identification to each log entry.
Interpreting web logs for compliance with “Acceptable Use” policy can be difficult. The web filter sees the original user request, each embedded web element and each overhead action performed by the web browser. Separating the log entries into those for which the user is, and is not, accountable is likely a matter of guesswork rather than certainty. In some configurations, the user attribution may also have inferences applied by the proxy as the data is being captured. Consequently, the organization must understand the integrity and limitations of its log data, then proceed cautiously when using the logs for employee discipline.
In any complex network, exceptions arise. Some web-enabled applications will be unable to authenticate with the proxy. Some applications or websites will use certificate pinning, which prevents HTTPS inspection. Some websites will misunderstand the browser configuration when the proxy is enabled. For these and other reasons, exceptions will be needed.
Web proxies allow these exceptions to be implemented on a centralized basis after appropriate management review and approval. Careful recordkeeping of these decisions is advised, because the exception configuration should be periodically audited, as the need for exceptions will change over time. Without supporting documentation, exceptions may accumulate unnecessarily and eventually introduce vulnerabilities.
Web proxies allow an organization to implement defenses that are impractical or impossible otherwise. Since web and email are the primary vehicles for recent malware and ransomware attacks, organizations can ill afford to ignore this important technology, or to tolerate a weak implementation of it.
Douglas Foster, CISSP, works in healthcare IT and lives in Virginia Beach, Va. This is his first article for Insights.
Carlos Cañoto is a virtual systems engineer manager originally from Caracas, Venezuela, and now working in Amsterdam, The Netherlands. He’s been an (ISC)2 member since 2006. An excerpt of this Q&A appears in the July/August issue of InfoSecurity Professional.
Did you ever think you’d be working in information security when you were a child?
Not really. My interest in computer science started at the beginning of high school (mid- to late ’80s) doing basic programming with “shy attempts” to develop video games, working with databases and information systems. The interest in the security field came as I started to see not only how programs worked, but also how they failed and how you can make them fail. Then after college, networking and security became my main interest.
What changed that led you to this career?
I wouldn’t call it a change. I would say curiosity is the main driver, the need to learn and the need to improve. From a professional and career point of view, security as a technology field has been in very high demand for professionals for as long as I can remember, which is a great opportunity for everyone interested.
What do you find most rewarding about what you do?
In my current role, I am not as involved in the day-to-day operations as I used to be (although I try to be as much as possible, as well as continuously learn and develop). But I am working closely with entry-level engineers and what I find rewarding is to support them as they gain the proper skills set and mindset for their future.
You recently moved from South America to The Netherlands. What was that transition like?
As with every change, it was difficult at the beginning: new culture, different ways of doing things, new regulations, new language, etc. But at the same time, it was, and still is, a wonderful experience, both from a professional and personal point of view. You have to learn new things every day. Also, as a father of two small children, I get to see them grow in such a rich, diverse environment. It’s amazing how fast they adapt and develop new skills.
What are you working on now that might improve information security in the near future?
At the moment, I think that helping to develop a new generation of engineers with the right set of tools and awareness about the field is the most meaningful contribution. Another project in mind for the future is the Center for Cyber Safety and Education’s Safe and Secure Online program, which helps kids start with the right foot in cyber education.
What solutions excite you, and why?
There are many, but currently, I would like to focus first on the security of “things.” We are quickly approaching a point where every common device is connecting to the network, which brings enormous value, but also increases the exposure to risks in a way we have never experienced before. It’s very difficult and there are many different approaches to the problem (depending on the kind of device, its criticality and even vendor), but I think every industry will (and many already are) put a significant effort towards securing their devices.
Where do you believe your peers in The Netherlands are experiencing the greatest threats to cyberattacks?
We live in a highly connected world, so the risks we face in The Netherlands are not very different than those faced in other parts of Europe or the United States. They also can come from anywhere in the world, but in general, I would say that a rapidly expanding attack surface is one big factor (related to the previous question). Now, the ability to consolidate data from many different sources and platforms, and being able to extract useful information in time to react, is more important than ever.
Do you ever miss your native country? What do you miss most about it?
Of course, you may leave your country, but you never leave your home. Friends and family are the most missed, as well as the beauty of living in a tropical country with wonderful sunny weather all year long and the spectacular landscapes that Venezuela offers.
If you could travel to anywhere to live, where would it be?
For now, we want to settle and enjoy our new country. In the future, we have always thought about living in Spain and that is a plan we still have, maybe in a few years.
And if someone were to travel to your native country, what should they be sure to see or do?
Venezuela is a country rich in nature. You can visit the south and experience the tropical forests; there you need to see the Angel Falls, the highest waterfall in the world almost 1km in height. Then move to the west and see the snowy mountains. Pico Bolivar, the highest one, is almost 5000m with permanent snow at the top. Another wonderful destination for a tropical country is the beach. The best ones are in the archipelago Los Roques, easily accessible from Caracas by a 20-minute plane flight.