使用Scala将文件写入Amazon S3 AWS存储

4szc88ey  于 2023-11-18  发布在  Scala
关注(0)|答案(4)|浏览(220)

这是我的代码的一部分:

val file=new File("s3a://tracce/output/Tempi.txt") 
   val writer=new BufferedWriter(new FileWriter(file,true))
   writer.write("Execution Time trace "+count.toString+"x"+i.toString+": "+differenza.toString)
   writer.newLine()
   writer.flush()
   writer.close()

字符串
我需要使用Scala向我的bucket中写入一个新文件。我得到以下错误:

Exception in thread "main" java.io.FileNotFoundException:     s3:/tracce/output/Tempi.txt (No such file or directory)


当我尝试在本地保存它时,它工作:

val file=new File("./Tempi.txt")


我该怎么解决呢?先谢了

nmpmafwu

nmpmafwu1#

您现在可以直接使用AWS Java SDK编写它。

import com.amazonaws.services.s3.{AmazonS3, AmazonS3ClientBuilder}

val s3Client: AmazonS3 = AmazonS3ClientBuilder.defaultClient()

val bucketName = "bucket-name"
val objectKey = "object-key"
val objectContent = "This is the file content."

s3Client.putObject(bucketName, objectKey, objectContent)

字符串

c0vxltue

c0vxltue2#

S3A是一个Hadoop文件系统客户端,你不能使用java File API来处理它。
1.创建一个hadoop Configuration对象,其中包含所有的s3凭证。如果您使用Spark,您可以从SparkContext免费获得。
1.使用Hadoop FS API获取FS示例,create()在其中创建文件。

import org.apache.hadoop.fs._
    import org.apache.hadoop.conf.Configuration
    val conf = new Configuration()
    conf.set("fs.s3a.access.key", "YOUR ACCESS KEY")
    conf.set("fs.s3a.secret.key", "YOUR SECRET KEY")

    val dest = new Path("s3a://bucket_name/output-path/file.txt")
    val fs = dest.getFileSystem(conf)
    val out = fs.create(dest, true)
    out.write( /* your string as bytes */ )
    out.close()

字符串

a0x5cqrl

a0x5cqrl3#

S3是一个对象存储器,你不能写字符串。你可以把字符串写到一个本地文件中,然后定期把文件上传到S3。

ctehm74n

ctehm74n4#

希望这对你有帮助。

def uploadDataToS3Bucket(baseDF: DataFrame): Unit = {
    // AWS S3 credentials and bucket information
    val accessKeyId = "YourAccessKey"
    val secretAccessKey = "YourSecretKey"
    val s3BucketName = "BucketName"
    val s3OutputPath = "path\filename.csv"
    val csvData = "id\tname\n1\tJohn\n2\tJane\n3\tAlice"
    
    // Set up AWS credentials
    System.setProperty("aws.accessKeyId", accessKeyId)
    System.setProperty("aws.secretKey", secretAccessKey)

    // Explicitly specify the AWS region
    val region = Regions.US_EAST_1

    // Create an AmazonS3 client
    val s3Client = AmazonS3ClientBuilder.standard()
      .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
      .withRegion(region.getName)
      .build()

    // Convert the CSV data to an InputStream
    val inputStream = new ByteArrayInputStream(csvData.getBytes)

    // Set the metadata for the object
    val metadata = new ObjectMetadata()
    metadata.setContentLength(csvData.getBytes.length)
    metadata.setContentType("text/csv")

    // Upload the data to S3
    val request = new PutObjectRequest(s3BucketName, s3OutputPath, inputStream, metadata)
    s3Client.putObject(request)

    println("Uploaded data to S3.")
  }

字符串

相关问题