如何使用ruby更新一批S3对象的元数据?

z6psavjg  于 2023-02-18  发布在  Ruby
关注(0)|答案(6)|浏览(164)

我需要修改S3上成百上千个对象的元数据(Content-Type)。使用ruby有什么好方法呢?据我所知,使用fog.io无法只保存元数据,整个对象必须重新保存。似乎使用the official sdk library需要我为这一个任务滚动 Package 器环境。

x7rlezfr

x7rlezfr1#

你说得对,官方SDK允许你修改对象元数据而无需再次上传。它所做的是复制对象,但对象在服务器上,所以你不需要下载文件并重新上传。
Package 器很容易实现,类似于

bucket.objects.each do |object|
  object.metadata['content-type'] = 'application/json'
end
njthzxwz

njthzxwz2#

在v2 API中,您可以使用Object#copy_from()Object.copy_to()以及:metadata:metadata_directive => 'REPLACE'选项来更新对象的元数据,而无需从S3下载。
Joost's gist中的代码抛出此错误:
Aws::S3::错误::无效请求:此复制请求是非法的,因为它试图将对象复制到自身,而不更改对象的元数据、存储类、网站重定向位置或加密属性。
这是因为默认情况下AWS会忽略复制操作提供的:metadata,因为它复制元数据,如果我们想就地更新元数据,我们必须设置:metadata_directive => 'REPLACE'选项。
请参见www.example.comhttp://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#copy_from-instance_method
下面是我最近用来执行元数据更新操作的一个完整的代码片段:

require 'aws-sdk'

# S3 setup boilerplate
client = Aws::S3::Client.new(
  :region => 'us-east-1',
  :access_key_id => ENV['AWS_ACCESS_KEY'],
  :secret_access_key => ENV['AWS_SECRET_KEY'], 
)
s3 = Aws::S3::Resource.new(:client => client)

# Get an object reference
object = s3.bucket('my-bucket-name').object('my-object/key')

# Create our new metadata hash. This can be any hash; in this example we update
# existing metadata with a new key-value pair.
new_metadata = object.metadata.merge('MY_NEW_KEY' => 'MY_NEW_VALUE')

# Use the copy operation to replace our metadata
object.copy_to(object,
  :metadata => new_metadata,

  # IMPORTANT: normally S3 copies the metadata along with the object.
  # we must supply this directive to replace the existing metadata with
  # the values we supply
  :metadata_directive => "REPLACE",
)

为便于重复使用:

def update_metadata(s3_object, new_metadata = {})
  s3_object.copy_to(s3_object,
    :metadata => new_metadata
    :metadata_directive => "REPLACE"
  )
end
cczfrluj

cczfrluj3#

对于未来的读者,这里有一个使用Ruby aws-sdk v1修改内容的完整示例(另请参见Gist中的aws-sdk v2示例):

# Using v1 of Ruby aws-sdk as currently v2 seems not able to do this (broken?).
require 'aws-sdk-v1'

key = YOUR_AWS_KEY
secret = YOUR_AWS_SECRET
region = YOUR_AWS_REGION

AWS.config(access_key_id: key, secret_access_key: secret, region: region)
s3 = AWS::S3.new
bucket = s3.buckets[bucket_name]
bucket.objects.with_prefix('images/').each do |obj|
  puts obj.key
  # Add  metadata: {} to next line for more metadata.
  obj.copy_from(obj.key, content_type: obj.content_type, cache_control: 'max-age=1576800000',  acl: :public_read)
end
ui7jx7zq

ui7jx7zq4#

经过一番搜索,这似乎对我有效

obj.copy_to(obj, :metadata_directive=>"REPLACE", :acl=>"public-read",:content_type=>"text/plain")
8wtpewkr

8wtpewkr5#

使用sdk来改变内容类型将导致x-amz-meta-前缀。我的解决方案是使用ruby + aws cli。这将直接写入content-type而不是x-amz-meta-content-type

ids_to_copy = all_object_ids
ids_to_copy.each do |id|
    object_key = "#{id}.pdf"
    command = "aws s3 cp s3://{bucket-name}/#{object_key} s3://{bucket-name}/#{object_key} --no-guess-mime-type --content-type='application/pdf' --metadata-directive='REPLACE'"
    system(command)
end
ltskdhd1

ltskdhd16#

此API现在似乎可用:

Fog::Storage.new({
  :provider                 => 'AWS',
  :aws_access_key_id        => 'foo',
  :aws_secret_access_key    => 'bar',
  :endpoint => 'https://s3.amazonaws.com/',
  :path_style => true
}).put_object_tagging(
  'bucket_name',
  's3_key',
  {foo: 'bar'}
)

相关问题