通过代理java下载文件

Question

我通过代理从www.example.com/example.pdf等网址下载文件时遇到问题，并将其保存在java文件系统中。有没有人有关于如何工作的想法？如果我得到的InputStream我可以简单地将它保存这个与文件系统：

final ReadableByteChannel rbc = Channels.newChannel(httpUrlConnetion.getInputStream());  
final FileOutputStream fos = new FileOutputStream(file); 
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE); 
fos.close(); 

但如何通过PROX获得一个网址的输入流？ 如果我这样做是这样的：

SocketAddress addr = new InetSocketAddress("my.proxy.com", 8080); 
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr); 
URL url = new URL("http://my.real.url.com/"); 
URLConnection conn = url.openConnection(proxy); 

我得到这个异常：使用

java.net.SocketException: Connection reset 
    at java.net.SocketInputStream.read(Unknown Source) 
    at java.net.SocketInputStream.read(Unknown Source) 
    at java.io.BufferedInputStream.fill(Unknown Source) 
    at java.io.BufferedInputStream.read1(Unknown Source) 
    at java.io.BufferedInputStream.read(Unknown Source) 
    at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) 
    at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) 
    at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) 
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) 
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) 
    at app.model.mail.crawler.newimpl.FileLoader.getSourceOfSiteViaProxy(FileLoader.java:167) 
    at app.model.mail.crawler.newimpl.FileLoader.process(FileLoader.java:220) 
    at app.model.mail.crawler.newimpl.FileLoader.run(FileLoader.java:57) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
    at java.lang.Thread.run(Unknown Source) 

这样的：

final HttpURLConnection httpUrlConnetion = (HttpURLConnection) website.openConnection(proxy); 
httpUrlConnetion.setDoOutput(true); 
httpUrlConnetion.setDoInput(true); 
httpUrlConnetion.setRequestProperty("Content-type", "text/xml"); 
httpUrlConnetion.setRequestProperty("Accept", "text/xml, application/xml"); 
httpUrlConnetion.setRequestMethod("POST"); 
httpUrlConnetion.connect(); 

我能够下载网站的来源，是HTML，但不是一个文件，也许有人可以帮助我与我必须设置为下载文件的属性。

Answer 1

以编程方式设置代理：

SocketAddress addr = new InetSocketAddress("my.proxy.com", 8080); 
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr); 
URL url = new URL("http://my.real.url.com/"); 
URLConnection conn = url.openConnection(proxy); 

然后你就可以使用上面与URLConnection在最后一行返回代码。如果您愿意，您也可以使用SOCKS代理，或者不强制代理。

这是从this Oracle documentation拍摄（并稍作编辑）。

Answer 2

可以使用Apache httpclient库解决与代理的大部分问题。要编译下面的代码，你可以使用以下Maven：

Maven的：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 

    <groupId>stackoverflow.test</groupId> 
    <artifactId>proxyhttp</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 
    <packaging>jar</packaging> 

    <name>proxy</name> 
    <url>http://maven.apache.org</url> 

    <properties> 
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
    </properties> 

    <dependencies> 
    <dependency> 
     <groupId>junit</groupId> 
     <artifactId>junit</artifactId> 
     <version>3.8.1</version> 
     <scope>test</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.httpcomponents</groupId> 
     <artifactId>httpclient</artifactId> 
     <version>4.5.1</version> 
    </dependency> 
    </dependencies> 
</project> 

Java代码：

import org.apache.http.HttpHost; 
import org.apache.http.client.config.RequestConfig; 
import org.apache.http.client.methods.CloseableHttpResponse; 
import org.apache.http.client.methods.HttpGet; 
import org.apache.http.impl.client.CloseableHttpClient; 
import org.apache.http.impl.client.HttpClients; 
import org.apache.http.util.EntityUtils; 

/** 
* How to send a request via proxy. 
* 
* @since 4.0 
*/ 
public class ClientExecuteProxy { 

    public static void main(String[] args)throws Exception { 
     CloseableHttpClient httpclient = HttpClients.createDefault(); 
     try { 
      HttpHost target = new HttpHost("www.google.com", 80, "http"); 
      HttpHost proxy = new HttpHost("127.0.0.1", 8889, "http"); 

      RequestConfig config = RequestConfig.custom() 
        .setProxy(proxy) 
        .build(); 
      HttpGet request = new HttpGet("/"); 
      request.setConfig(config); 

      System.out.println("Executing request " + request.getRequestLine() + " to " + target + " via " + proxy); 

      CloseableHttpResponse response = httpclient.execute(target, request); 
      try { 
       System.out.println("----------------------------------------"); 
       System.out.println(response.getStatusLine()); 
       System.out.println(EntityUtils.toString(response.getEntity())); 
      } finally { 
       response.close(); 
      } 
     } finally { 
      httpclient.close(); 
     } 
    } 

} 

Answer 3

以下是从其他答案不同，对我的作品：集这些属性连接之前：

  System.getProperties().put("http.proxySet", "true"); 
      System.getProperties().put("http.proxyHost", "my.proxy.com"); 
      System.getProperties().put("http.proxyPort", "8080"); //port is String, not int 

然后，打开URLConnection和tr ❖下载文件。

Answer 4

另一种方法是在httpUrlConnection的每个实例“内部”实现代理。那就是：

1. 不要连接到你想要的真实URL。首先，连接到代理IP和端口，但使用http GET方法引用您想要的URL。
2. 使用setRequestProperty将主机设置为您的URL和您可能需要的任何其他标头。

如果有效，连接将透明地将文件发送给您。

我有一些与套接字一起工作的代码。

try { 
    Socket sock = new Socket("10.0.241.1", 3128); //proxy IP and port 
    InputStream is = sock.getInputStream(); 
    OutputStream os = sock.getOutputStream(); 
    String str = "GET http://www.uol.com.br HTTP/1.1\r\n"; //GET your site 
    str += "Host: www.uol.com.br\r\n"; //again, Host of your site 
    str += "Proxy-Authorization: Basic ZWR1YXJkby5wb2NvOmM1NmQyMw==\r\n"; //if password is needed 
    str += "\r\n"; 
    os.write(str.getBytes()); 
    byte[] bb = new byte[1024]; 
    int L = 0; 
    while ((L = is.read(bb)) != -1) { 
     //write bytes to file stream... 
    } 
} catch (Exception ex) { 
    //exception handling... 
} 

“为什么有人在使用httpUrlConnection时使用纯套接字？”，你说。那么，那个时候，我不知道httpUrlConnection。