从亚马逊s3流中读取文本

Question

我使用以下代码从Amazon S3读取文本文件，并逐行处理它。此代码有效，但问题是速度很慢。

GetObjectRequest getObjRequest = new GetObjectRequest() 
    .WithBucketName(amazonSettings.BucketName) 
    .WithKey(_fileKey); 
using (AmazonS3 client = AWSClientFactory.CreateAmazonS3Client(
    amazonSettings.AccessKey, 
    amazonSettings.SecretAccessKey)) 
using (GetObjectResponse getObjRespone = client.GetObject(getObjRequest)) 
using (Stream amazonStream = getObjRespone.ResponseStream) 
{       
    StreamReader amazonStreamReader = new StreamReader(amazonStream); 
    tempGsContact = new GSContact(); 
    while ((_fileLine = amazonStreamReader.ReadLine()) != null) 
    { 
     if (_fileLine.Equals("END:VCARD")) 
     { 
      // Make process 1 
     } 
     else if (!_fileLine.Equals(string.Empty)) 
     { 
      //Make process 2 
     } 
    }       
} 

问题是：我能否获得更充分的方法来降低时间成本？

Answer 1

.NET中的HTTPWebResponse存在类似的性能瓶颈，这可能是他们制作的AmazonS3类所包装的。

它是由花费很长的时间来解决代理设置对象造成的，也有上市here一些潜在的解决方案，但最简单的选择可能是以下添加到您的app.config文件：

<system.net> 
    <defaultProxy enabled="false"> 
    <proxy/> 
    <bypasslist/> 
    <module/> 
    </defaultProxy> 
</system.net> 

一起接受“AmazonS3Config”的第三个参数，你可以指定通过“AmazonS3Config.ProxyHost = NULL”零代理的 overload通话

AWSClientFactory.CreateAmazonS3Client(amazonSettings.AccessKey, amazonSettings.SecretAccessKey) 

- 这应该：

另外，您可以在这里调用替换实际上与仅针对该请求的上述配置更改相同。

Answer 2

该函数用于计算文本文件中的字数。

private int WordCount(string awsBucketName, string awsFilePath, string wordForSearch) 
    { 
     string line =string.Empty; 
     int counter = 0; 
     if ((cloudKaseClient != null) & (token == tokenCfg)) 
     { 
      var request = new GetObjectRequest() 
      { 
       BucketName = awsBucketName, 
       Key = awsFilePath 
      }; 
      using (var response = cloudKaseClient.GetObject(request)) 
      { 
        StreamReader reader = new StreamReader(response.ResponseStream); 
        while ((line = reader.ReadLine()) != null) 
        { 
         counter += line.Split(' ').Where(t => t.ToString().ToLower().Contains(wordForSearch.ToLower())).Count(); 
        } 
        reader.Close();    
      } 
     } 
     return counter; 
    } 

Answer 3

，或者您可以使用此功能还，

private static void ReadS3Object(AmazonS3Client client) 
    { 
     GetObjectRequest request = new GetObjectRequest(); 

     request.WithBucketName(BUCKET_NAME); 
     request.WithKey(S3_KEY); 

     GetObjectResponse response = client.GetObject(request); 

     StreamReader reader = new StreamReader(response.ResponseStream); 

     String content = reader.ReadToEnd(); 

     Console.Out.WriteLine("Read S3 object with key " + S3_KEY + " in bucket " + BUCKET_NAME + ". Content is: " + content); 
    }