org.deeplearning4j.aws.s3.reader

Class S3Downloader

java.lang.Object

org.deeplearning4j.aws.s3.BaseS3

org.deeplearning4j.aws.s3.reader.S3Downloader

public class S3Downloader
extends BaseS3

Downloads files from S3

Field Summary

Fields inherited from class org.deeplearning4j.aws.s3.BaseS3

ACCESS_KEY, ACCESS_SECRET, accessKey, AWS_ACCESS_KEY, AWS_SECRET_KEY, creds, secretKey, serialVersionUID

Constructor Summary

Constructors

Constructor and Description
S3Downloader()

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	buckets() Returns the list of buckets in s3
void	download(java.lang.String bucket, java.lang.String key, java.io.File to)
void	download(java.lang.String bucket, java.lang.String key, java.io.OutputStream to)
com.amazonaws.services.s3.transfer.MultipleFileDownload	downloadFolder(java.lang.String bucketName, java.lang.String keyPrefix, java.io.File folderPath)
java.util.Iterator<java.io.InputStream>	iterateBucket(java.lang.String bucket) Iterate over individual buckets.
java.util.List<java.lang.String>	keysForBucket(java.lang.String bucket) Return the keys for a bucket
com.amazonaws.services.s3.model.ObjectListing	listObjects(java.lang.String bucket) Simple way of retrieving the listings for a bucket
com.amazonaws.services.s3.model.ObjectListing	nextList(com.amazonaws.services.s3.model.ObjectListing list) Iterator style one list at a time
java.io.InputStream	objectForKey(java.lang.String bucket, java.lang.String key) Returns an input stream for the given bucket and key
void	paginate(java.lang.String bucket, BucketKeyListener listener) Paginates through a bucket's keys invoking the listener at each key

Methods inherited from class org.deeplearning4j.aws.s3.BaseS3

findCreds, getClient, getCreds, getEc2, setCreds

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3Downloader

public S3Downloader()

Method Detail

keysForBucket

public java.util.List<java.lang.String> keysForBucket(java.lang.String bucket)

Return the keys for a bucket

Parameters:

bucket - the bucket to get the keys for

Returns:

the bucket's keys

buckets

public java.util.List<java.lang.String> buckets()

Returns the list of buckets in s3

Returns:

the list of buckets

iterateBucket

public java.util.Iterator<java.io.InputStream> iterateBucket(java.lang.String bucket)

Iterate over individual buckets. Returns input streams to each object. It is your responsibility to close the input streams

Parameters:

bucket - the bucket to iterate over

Returns:

an iterator over the objects in an s3 bucket

nextList

public com.amazonaws.services.s3.model.ObjectListing nextList(com.amazonaws.services.s3.model.ObjectListing list)

Iterator style one list at a time

Parameters:

list - the list to getFromOrigin the next batch for

Returns:

the next batch of objects or null if none are left

listObjects

public com.amazonaws.services.s3.model.ObjectListing listObjects(java.lang.String bucket)

Simple way of retrieving the listings for a bucket

Parameters:

bucket - the bucket to retrieve listings for

Returns:

the object listing for this bucket

paginate

public void paginate(java.lang.String bucket,
                     BucketKeyListener listener)

Paginates through a bucket's keys invoking the listener at each key

Parameters:

bucket - the bucket to iterate

listener - the listener

objectForKey

public java.io.InputStream objectForKey(java.lang.String bucket,
                                        java.lang.String key)

Returns an input stream for the given bucket and key

Parameters:

bucket - the bucket to retrieve from

key - the key of the objec t

Returns:

an input stream to the object

download

public void download(java.lang.String bucket,
                     java.lang.String key,
                     java.io.File to)
              throws java.io.IOException

Throws:

java.io.IOException

download

public void download(java.lang.String bucket,
                     java.lang.String key,
                     java.io.OutputStream to)
              throws java.io.IOException

Throws:

java.io.IOException

downloadFolder

public com.amazonaws.services.s3.transfer.MultipleFileDownload downloadFolder(java.lang.String bucketName,
                                                                              java.lang.String keyPrefix,
                                                                              java.io.File folderPath)