Amazon Glacier并行存档上传

我有一个示例应用程序，它使用AWS SDK for .NET高级API将大小为26 MB的文件上传到Amazon Glacier。该代码工作正常，不必穿线但线程池的失败在下面的行Amazon Glacier并行存档上传

  client.UploadMultipartPart(uploadMPUrequest);

说法错误消息：请求已中止：请求已被取消。

堆栈跟踪：在Amazon.Runtime.AmazonWebServiceClient.handleHttpWebErrorResponse（AsyncResult asyncResult，引发WebException我们）在Amazon.Runtime.AmazonWebServiceClient.getRequestStreamCallback（IAsyncResult的结果）在Amazon.Runtime.AmazonWebServiceClient.InvokeConfiguredRequest（AsyncResult asyncResult）在Amazon.Runtime.AmazonWebServiceClient.InvokeHelper（asyncResult asyncResult）在Amazon.Runtime.AmazonWebServiceClient.Invoke（asyncResult asyncResult）在Amazon.Glacier.AmazonGlacierClient.invokeUploadMultipartPart亚马逊（UploadMultipartPartRequest uploadMultipartPartRequest，AsyncCallback的回调，对象的状态，布尔同步）。 Glacier.AmazonGlacierClient.UploadMultipartPa RT（UploadMultipartPartRequest uploadMultipartPartRequest）

注：我上传的多部分数据

请找到下面的链接，我的示例代码： www.page-monitor.com/Downloads/ArchiveUploadMPU.cs

是否有存档的并行上传任何示例代码？

感谢和问候， Haseena

来源

2013-04-18 Haseena Parkar

谷歌搜索发现这个git项目，也许它会回答你的问题：https：//github.com/athomason/ParallelGlacierUploader编辑：哦只是注意到这是一个C＃的问题，没关系 – RuntimeError

嘿谢谢，这实际上帮助了我。现在我可以并行上传档案到冰川。再次感谢！！ –

没问题，google是你的朋友;） – RuntimeError

以下是可与线程正常工作的示例代码。 ChunkDetails是一个用于传递accessID，bucketname，offset详细信息等的自定义库。我也使用ThrottledStream。

 internal bool UploadUsingHighLevelAPI(String FilePath, ChunkDetails ObjMetaData, 
              S3Operations.UploadType uploadType, 
     Stream inputStream) 
     { 
     String METHOD_NAME = "UploadUsingHighLevelAPI"; 
     String keyName; 
     String existingBucketName; 
     TransferUtilityUploadRequest fileTransferUtilityRequest = null; 
     int RetryTimes = 3; 
     ThrottledStream throttleStreamObj = null; 

     long bps = ThrottledStream.Infinite; 
     try 
     { 


      keyName = ObjMetaData.KeyName; 
      existingBucketName = ObjMetaData.BucketName; 

      TransferUtility fileTransferUtility = new 
        TransferUtility(ObjMetaData.AccessKeyID,  ObjMetaData.SecretAccessKey); 

      FileInfo fin = new FileInfo(FilePath); 

      //streamObj = new FileStream(FilePath, FileMode.Open); 

      bps = (long)(1024 * ObjMetaData.MaxAvailSpeed * ((double)ObjMetaData.Bandwidth/100.0)); 


      throttleStreamObj = new ThrottledStream(ObjMetaData.FileStream, bps); 


      System.Collections.Specialized.NameValueCollection metaInfo = new System.Collections.Specialized.NameValueCollection(); 
      if (ObjMetaData.MetaInfo != null) 
      { 
       foreach (DictionaryEntry kvp in ObjMetaData.MetaInfo) 
       { 
        metaInfo.Add(kvp.Key.ToString(), kvp.Value.ToString()); 
       } 
      } 


      long OffDiff = ObjMetaData.EndOffset - ObjMetaData.StartOffset; 
      long partSize; 
      if (fin.Length >= OffDiff) 
      { 
       partSize = OffDiff; 
      } 
      else 
       partSize = fin.Length; 



      if (uploadType == UploadType.File) 
      { 
       //fileTransferUtility.Upload(FilePath, existingBucketName, keyName); 


       fileTransferUtilityRequest = 
       new TransferUtilityUploadRequest() 
       .WithBucketName(existingBucketName) 
       //.WithFilePath(FilePath) 
       .WithStorageClass(S3StorageClass.ReducedRedundancy) 
       .WithMetadata(metaInfo) 
       .WithPartSize(partSize) 
       .WithKey(keyName) 
       .WithCannedACL(S3CannedACL.PublicRead) 
       .WithTimeout(Int32.MaxValue - 1) 
       .WithInputStream(throttleStreamObj) as TransferUtilityUploadRequest; 

      } 
      else if (uploadType == UploadType.Stream) 
      { 
       fileTransferUtilityRequest = 
       new TransferUtilityUploadRequest() 
       .WithBucketName(existingBucketName) 
       .WithStorageClass(S3StorageClass.ReducedRedundancy) 
       .WithMetadata(metaInfo) 
       .WithPartSize(partSize) 
       .WithKey(keyName) 
       .WithCannedACL(S3CannedACL.PublicRead) 
       .WithTimeout(Int32.MaxValue - 1) 
       .WithInputStream(throttleStreamObj) as TransferUtilityUploadRequest 
       ; 
      } 



      for (int index = 1; index <= RetryTimes; index++) 
      { 
       try 
       { 

        // Upload part and add response to our list. 
        fileTransferUtility.Upload(fileTransferUtilityRequest); 
        Console.WriteLine(" ====== Upload Done ========="); 
        if (eventChunkUploaded != null) 
         eventChunkUploaded(ObjMetaData); 
        break; 

       } 
       catch (Exception ex) 
       { 
        if (index == RetryTimes) 
        { 
         m_objLogFile.LogError(CLASS_NAME, METHOD_NAME + " - Attempt " + 
          index + Environment.NewLine + FilePath, ex); 

         if (eventChunkUploadError != null) 
          eventChunkUploadError(ObjMetaData, ex.Message); 

        } 
       } 
      } 
     } 
     catch (Exception ex) 
     { 
      m_objLogFile.LogError(CLASS_NAME, METHOD_NAME, ex); 
      return false; 
     } 
     finally 
     { 

      if (throttleStreamObj != null) 
      { 
       //inputStream1.Close(); 
       throttleStreamObj = null; 
      } 
     } 

     return true; 
    }

让我知道，如果你面对的任何问题。

来源

2013-10-03 05:29:52

我相信这是在代码中的竞争条件。我正在研究相同的功能。我很乐意与您分享代码。如果你修正了你发布的代码，我会很感激链接。最好的问候，布鲁斯

来源

2013-09-30 15:47:06

嗨，布鲁斯，请参考贴子的答复。 –

Amazon Glacier并行存档上传

回答

相关问题