Package org.htmlunit

Class HttpWebConnection

  • All Implemented Interfaces:
    java.lang.AutoCloseable, WebConnection

    public class HttpWebConnection
    extends java.lang.Object
    implements WebConnection
    Default implementation of WebConnection, using the HttpClient library to perform HTTP requests.
    • Field Detail

      • LOG

        private static final org.apache.commons.logging.Log LOG
      • HACKED_COOKIE_POLICY

        private static final java.lang.String HACKED_COOKIE_POLICY
        See Also:
        Constant Field Values
      • httpClientBuilder_

        private final java.util.Map<java.lang.Thread,​org.apache.http.impl.client.HttpClientBuilder> httpClientBuilder_
      • webClient_

        private final WebClient webClient_
      • virtualHost_

        private java.lang.String virtualHost_
      • connectionManager_

        private org.apache.http.impl.conn.PoolingHttpClientConnectionManager connectionManager_
      • sharedAuthCache_

        private final org.apache.http.client.AuthCache sharedAuthCache_
        Authentication cache shared among all threads of a web client.
      • httpClientContextByThread_

        private final java.util.Map<java.lang.Thread,​org.apache.http.client.protocol.HttpClientContext> httpClientContextByThread_
        Maintains a separate HttpClientContext object per HttpWebConnection and thread.
    • Constructor Detail

      • HttpWebConnection

        public HttpWebConnection​(WebClient webClient)
        Creates a new HTTP web connection instance.
        Parameters:
        webClient - the WebClient that is using this connection
    • Method Detail

      • getResponse

        public WebResponse getResponse​(WebRequest webRequest)
                                throws java.io.IOException
        Submits a request and retrieves a response.
        Specified by:
        getResponse in interface WebConnection
        Parameters:
        webRequest - the request
        Returns:
        the response to the request defined by the specified request
        Throws:
        java.io.IOException - if an IO error occurs
      • onResponseGenerated

        protected void onResponseGenerated​(org.apache.http.client.methods.HttpUriRequest httpMethod)
        Called when the response has been generated. Default action is to release the HttpMethod's connection. Subclasses may override.
        Parameters:
        httpMethod - the httpMethod used (can be null)
      • getHttpContext

        private org.apache.http.protocol.HttpContext getHttpContext()
        Returns the HttpClientContext for the current thread. Creates a new one if necessary.
      • setProxy

        private void setProxy​(org.apache.http.client.methods.HttpRequestBase httpRequest,
                              WebRequest webRequest)
      • makeHttpMethod

        private org.apache.http.client.methods.HttpUriRequest makeHttpMethod​(WebRequest webRequest,
                                                                             org.apache.http.impl.client.HttpClientBuilder httpClientBuilder)
                                                                      throws java.net.URISyntaxException
        Creates an HttpMethod instance according to the specified parameters.
        Parameters:
        webRequest - the request
        httpClientBuilder - the httpClientBuilder that will be configured
        Returns:
        the HttpMethod instance constructed according to the specified parameters
        Throws:
        java.net.URISyntaxException - in case of syntax problems
      • escapeQuery

        private static java.lang.String escapeQuery​(java.lang.String query)
      • getCharset

        private static java.nio.charset.Charset getCharset​(java.nio.charset.Charset charset,
                                                           java.util.List<NameValuePair> pairs)
      • buildFilePart

        void buildFilePart​(KeyDataPair pairWithFile,
                           org.apache.http.entity.mime.MultipartEntityBuilder builder)
      • buildHttpMethod

        private static org.apache.http.client.methods.HttpRequestBase buildHttpMethod​(HttpMethod submitMethod,
                                                                                      java.net.URI uri)
        Creates and returns a new HttpClient HTTP method based on the specified parameters.
        Parameters:
        submitMethod - the submit method being used
        uri - the uri being used
        Returns:
        a new HttpClient HTTP method based on the specified parameters
      • getHttpClientBuilder

        protected org.apache.http.impl.client.HttpClientBuilder getHttpClientBuilder()
        Lazily initializes the internal HTTP client.
        Returns:
        the initialized HTTP client
      • getTimeout

        protected int getTimeout​(WebRequest webRequest)
        Returns the timeout to use for socket and connection timeouts for HttpConnectionManager. Is overridden to 0 by StreamingWebConnection which keeps reading after a timeout and must have long running connections explicitly terminated.
        Parameters:
        webRequest - the request might have his own timeout
        Returns:
        the WebClient's timeout
      • createHttpClientBuilder

        protected org.apache.http.impl.client.HttpClientBuilder createHttpClientBuilder()
        Creates the HttpClientBuilder that will be used by this WebClient. Extensions may override this method in order to create a customized HttpClientBuilder instance (e.g. with a custom ClientConnectionManager to perform some tracking; see feature request 1438216).
        Returns:
        the HttpClientBuilder that will be used by this WebConnection
      • configureTimeout

        private void configureTimeout​(org.apache.http.impl.client.HttpClientBuilder builder,
                                      int timeout)
      • createRequestConfigBuilder

        private static org.apache.http.client.config.RequestConfig.Builder createRequestConfigBuilder​(int timeout,
                                                                                                      java.net.InetAddress localAddress)
      • createSocketConfigBuilder

        private static org.apache.http.config.SocketConfig.Builder createSocketConfigBuilder​(int timeout)
      • reconfigureHttpClientIfNeeded

        private org.apache.http.impl.client.HttpClientBuilder reconfigureHttpClientIfNeeded​(org.apache.http.impl.client.HttpClientBuilder httpClientBuilder,
                                                                                            WebRequest webRequest)
        React on changes that may have occurred on the WebClient settings. Registering as a listener would be probably better.
      • configureHttpsScheme

        private void configureHttpsScheme​(org.apache.http.impl.client.HttpClientBuilder builder)
      • configureHttpProcessorBuilder

        private void configureHttpProcessorBuilder​(org.apache.http.impl.client.HttpClientBuilder builder,
                                                   WebRequest webRequest)
      • setVirtualHost

        public void setVirtualHost​(java.lang.String virtualHost)
        Sets the virtual host.
        Parameters:
        virtualHost - the virtualHost to set
      • getVirtualHost

        public java.lang.String getVirtualHost()
        Gets the virtual host.
        Returns:
        virtualHost The current virtualHost
      • makeWebResponse

        protected WebResponse makeWebResponse​(org.apache.http.HttpResponse httpResponse,
                                              WebRequest webRequest,
                                              DownloadedContent responseBody,
                                              long loadTime)
        Converts an HttpMethod into a WebResponse.
        Parameters:
        httpResponse - the web server's response
        webRequest - the WebRequest
        responseBody - the DownloadedContent
        loadTime - the download time
        Returns:
        a wrapper for the downloaded body.
      • downloadResponse

        protected WebResponse downloadResponse​(org.apache.http.client.methods.HttpUriRequest httpMethod,
                                               WebRequest webRequest,
                                               org.apache.http.HttpResponse httpResponse,
                                               long startTime)
                                        throws java.io.IOException
        Downloads the response. This calls downloadResponseBody(HttpResponse) and constructs the WebResponse.
        Parameters:
        httpMethod - the HttpUriRequest
        webRequest - the WebRequest
        httpResponse - the web server's response
        startTime - the download start time
        Returns:
        a wrapper for the downloaded body.
        Throws:
        java.io.IOException - in case of problem reading/saving the body
      • downloadResponseBody

        protected DownloadedContent downloadResponseBody​(org.apache.http.HttpResponse httpResponse)
                                                  throws java.io.IOException
        Downloads the response body.
        Parameters:
        httpResponse - the web server's response
        Returns:
        a wrapper for the downloaded body.
        Throws:
        java.io.IOException - in case of problem reading/saving the body
      • downloadContent

        public static DownloadedContent downloadContent​(java.io.InputStream is,
                                                        int maxInMemory,
                                                        java.io.File tempFileDirectory)
                                                 throws java.io.IOException
        Reads the content of the stream and saves it in memory or on the file system.
        Parameters:
        is - the stream to read
        maxInMemory - the maximumBytes to store in memory, after which save to a local file
        tempFileDirectory - the directory to be used or null for the system default
        Returns:
        a wrapper around the downloaded content
        Throws:
        java.io.IOException - in case of read issues
      • newWebResponseInstance

        protected WebResponse newWebResponseInstance​(WebResponseData responseData,
                                                     long loadTime,
                                                     WebRequest webRequest)
        Constructs an appropriate WebResponse. May be overridden by subclasses to return a specialized WebResponse.
        Parameters:
        responseData - Data that was send back
        webRequest - the request used to get this response
        loadTime - How long the response took to be sent
        Returns:
        the new WebResponse
      • getHttpRequestInterceptors

        private java.util.List<org.apache.http.HttpRequestInterceptor> getHttpRequestInterceptors​(WebRequest webRequest)
      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface WebConnection
      • createConnectionManager

        private static org.apache.http.impl.conn.PoolingHttpClientConnectionManager createConnectionManager​(org.apache.http.impl.client.HttpClientBuilder builder)
        Has the exact logic in HttpClientBuilder.build() which sets the connManager part, but with the ability to configure socketFactory.
      • split

        private static java.lang.String[] split​(java.lang.String s)
      • getField

        private static <T> T getField​(java.lang.Object target,
                                      java.lang.String fieldName)
                               throws java.lang.IllegalAccessException
        Throws:
        java.lang.IllegalAccessException