如何翻译的AWS S3网址成博托桶的名字吗？

Question

我试图与博托访问http://s3.amazonaws.com/commoncrawl/parse-output/segment/桶。我无法弄清楚如何将其转换为boto.s3.bucket.Bucket（）的名称。

这就是我要去为我的要点：

s3 = boto.connect_s3() 
cc = boto.s3.bucket.Bucket(connection=s3, name='commoncrawl/parse-output/segment') 
requester = {'x-amz-request-payer':'requester'} 
contents = cc.list(headers=requester) 
for i,item in enumerate(contents): 
    print item.__repr__() 

我得到“boto.exception.S3ResponseError：S3ResponseError：400错误的请求......指定桶无效...”

Answer 1

桶名称将是commoncrawl。之后出现的所有内容实际上只是存储桶中出现的密钥名称的一部分。

Answer 2

的AWS documents list four possible url formats for S3 - 这里的东西我只是把一起提取桶和地区所有的不同的URL格式。

import re 

def bucket_name_from_url(url): 
    """ Gets bucket name and region from url, matching any of the different formats for S3 urls 
    * http://bucket.s3.amazonaws.com 
    * http://bucket.s3-aws-region.amazonaws.com 
    * http://s3.amazonaws.com/bucket 
    * http://s3-aws-region.amazonaws.com/bucket 

    returns bucket name, region 
    """  
    match = re.search('^https?://([^.]+).s3.amazonaws.com/', url) 
    if match: 
     return match.group(1), None 

    match = re.search('^https?://([^.]+).s3-([^.]+).amazonaws.com/', url) 
    if match: 
     return match.group(1), match.group(2) 

    match = re.search('^https?://s3.amazonaws.com/([^\/]+)', url) 
    if match: 
     return match.group(1), None 

    match = re.search('^https?://s3-([^.]+).amazonaws.com/([^\/]+)', url) 
    if match: 
     return match.group(2), match.group(1) 

    return None, None 

像这样的东西确实应该进入博托......亚马逊，我希望你在听

Answer 3

基础上马克的答案，我做了一个小pyparsing脚本更清晰，我（包括可能的关键比赛）：

#!/usr/bin/env python 

from pyparsing import Word, alphanums, Or, Optional, Combine 

schema = Or(['http://', 'https://']).setResultsName('schema') 
word = Word(alphanums + '-', min=1) 
bucket_name = word.setResultsName('bucket') 
region = word.setResultsName('region') 

key = Optional('/' + word.setResultsName('key')) 

"bucket.s3.amazonaws.com" 
opt1 = Combine(schema + bucket_name + '.s3.amazonaws.com' + key) 

"bucket.s3-aws-region.amazonaws.com" 
opt2 = Combine(schema + bucket_name + '.' + region + '.amazonaws.com' + key) 

"s3.amazonaws.com/bucket" 
opt3 = Combine(schema + 's3.amazonaws.com/' + bucket_name + key) 

"s3-aws-region.amazonaws.com/bucket" 
opt4 = Combine(schema + region + ".amazonaws.com/" + bucket_name + key) 

tests = [ 
    "http://bucket-name.s3.amazonaws.com", 
    "https://bucket-name.s3-aws-region-name.amazonaws.com", 
    "http://s3.amazonaws.com/bucket-name", 
    "https://s3-aws-region-name.amazonaws.com/bucket-name", 
    "http://bucket-name.s3.amazonaws.com/key-name", 
    "https://bucket-name.s3-aws-region-name.amazonaws.com/key-name", 
    "http://s3.amazonaws.com/bucket-name/key-name", 
    "https://s3-aws-region-name.amazonaws.com/bucket-name/key-name", 
] 

s3_url = Or([opt1, opt2, opt3, opt4]).setResultsName('url') 

for test in tests: 
    result = s3_url.parseString(test) 
    print "found url: " + str(result.url) 
    print "schema: " + str(result.schema) 
    print "bucket name: " + str(result.bucket) 
    print "key name: " + str(result.key) 

原来我做了马克的脚本还检索键（对象）：

def parse_s3_url(url): 
    """ Gets bucket name and region from url, matching any of the different formats for S3 urls 
    * http://bucket.s3.amazonaws.com 
    * http://bucket.s3-aws-region.amazonaws.com 
    * http://s3.amazonaws.com/bucket 
    * http://s3-aws-region.amazonaws.com/bucket 

    returns bucket name, region 
    """ 
    match = re.search('^https?://([^.]+).s3.amazonaws.com(/\([^.]+\))', url) 
    if match: 
     return match.group(1), None, match.group(2) 

    match = re.search('^https?://([^.]+).s3-([^.]+).amazonaws.com/', url) 
    if match: 
     return match.group(1), match.group(2), match.group(3) 

    match = re.search('^https?://s3.amazonaws.com/([^\/]+)', url) 
    if match: 
     return match.group(1), None, match.group(2) 

    match = re.search('^https?://s3-([^.]+).amazonaws.com/([^\/]+)', url) 
    if match: 
     return match.group(2), match.group(1), match.group(3) 

    return None, None, None 

Answer 4

扩展标记回答返回键

#!/usr/bin/env python 

import re 

def parse_s3_url(url): 
    # returns bucket_name, region, key 

    bucket_name = None 
    region = None 
    key = None 

    # http://bucket.s3.amazonaws.com/key1/key2 
    match = re.search('^https?://([^.]+).s3.amazonaws.com(.*?)$', url) 
    if match: 
     bucket_name, key = match.group(1), match.group(2) 

    # http://bucket.s3-aws-region.amazonaws.com/key1/key2 
    match = re.search('^https?://([^.]+).s3-([^\.]+).amazonaws.com(.*?)$', url) 
    if match: 
     bucket_name, region, key = match.group(1), match.group(2), match.group(3) 

    # http://s3.amazonaws.com/bucket/key1/key2 
    match = re.search('^https?://s3.amazonaws.com/([^\/]+)(.*?)$', url) 
    if match: 
     bucket_name, key = match.group(1), match.group(2) 

    # http://s3-aws-region.amazonaws.com/bucket/key1/key2 
    match = re.search('^https?://s3-([^.]+).amazonaws.com/([^\/]+)(.*?)$', url) 
    if match: 
     bucket_name, region, key = match.group(2), match.group(1), match.group(3) 

    return list(map(lambda x: x.strip('/') if x else None, [bucket_name, region, key])) 

Answer 5

这是我的JS版本：

function parseS3Url(url) { 
    // Process all aws s3 url cases 

    url = decodeURIComponent(url); 
    let match = ""; 

    // http://s3.amazonaws.com/bucket/key1/key2 
    match = url.match(/^https?:\/\/s3.amazonaws.com\/([^\/]+)\/?(.*?)$/); 
    if (match) { 
    return { 
     bucket: match[1], 
     key: match[2], 
     region: "" 
    }; 
    } 

    // http://s3-aws-region.amazonaws.com/bucket/key1/key2 
    match = url.match(/^https?:\/\/s3-([^.]+).amazonaws.com\/([^\/]+)\/?(.*?)$/); 
    if (match) { 
    return { 
     bucket: match[2], 
     key: match[3], 
     region: match[1] 
    }; 
    } 

    // http://bucket.s3.amazonaws.com/key1/key2 
    match = url.match(/^https?:\/\/([^.]+).s3.amazonaws.com\/?(.*?)$/); 
    if (match) { 
    return { 
     bucket: match[1], 
     key: match[2], 
     region: "" 
    }; 
    } 

    // http://bucket.s3-aws-region.amazonaws.com/key1/key2 
    match = url.match(/^https?:\/\/([^.]+).s3-([^\.]+).amazonaws.com\/?(.*?)$/); 
    if (match) { 
    return { 
     bucket: match[1], 
     key: match[3], 
     region: match[2] 
    }; 
    } 

    return { 
    bucket: "", 
    key: "", 
    region: "" 
    }; 
}