Fix S3Session for pagination
According with AWS SDK doc the nextMarker field is only returned when defining the delimiter query param:
> NextMarker
When the response is truncated (the IsTruncated element value in the response is true), you can use the key name in this field as the marker parameter in the subsequent request to get the next set of objects. Amazon S3 lists objects in alphabetical order.
Note
This element is returned only if you have the delimiter request parameter specified. If the response does not include the NextMarker element and it is truncated, you can use the value of the last Key element in the response as the marker parameter in the subsequent request to get the next set of object keys.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_ResponseElements
After upgrading my project to newest version of `spring-integration-aws` I've started facing this loop issue for buckets with more than `1000` items, where `isTruncated` is true and `nextMarker` is `null`.
* Update the `S3Session` code to use the `key` from last item as `nextMarker` in next the request when response `isTruncated`, avoiding infinite loop.
This commit is contained in:
@@ -48,6 +48,7 @@ import org.springframework.util.StringUtils;
|
||||
* @author Jim Krygowski
|
||||
* @author Anwar Chirakkattil
|
||||
* @author Xavier François
|
||||
* @author Rogerio Lino
|
||||
*/
|
||||
public class S3Session implements Session<S3Object> {
|
||||
|
||||
@@ -82,8 +83,11 @@ public class S3Session implements Session<S3Object> {
|
||||
List<S3Object> objectSummaries = new ArrayList<>();
|
||||
do {
|
||||
objectListing = this.amazonS3.listObjects(listObjectsRequest.build());
|
||||
objectSummaries.addAll(objectListing.contents());
|
||||
listObjectsRequest.marker(objectListing.nextMarker());
|
||||
List<S3Object> contents = objectListing.contents();
|
||||
objectSummaries.addAll(contents);
|
||||
if (objectListing.isTruncated()) {
|
||||
listObjectsRequest.marker(contents.get(contents.size() - 1).key());
|
||||
}
|
||||
}
|
||||
while (objectListing.isTruncated());
|
||||
|
||||
@@ -108,10 +112,13 @@ public class S3Session implements Session<S3Object> {
|
||||
List<String> names = new ArrayList<>();
|
||||
do {
|
||||
objectListing = this.amazonS3.listObjects(listObjectsRequest.build());
|
||||
for (S3Object objectSummary : objectListing.contents()) {
|
||||
List<S3Object> contents = objectListing.contents();
|
||||
for (S3Object objectSummary : contents) {
|
||||
names.add(objectSummary.key());
|
||||
}
|
||||
listObjectsRequest.marker(objectListing.nextMarker());
|
||||
if (objectListing.isTruncated()) {
|
||||
listObjectsRequest.marker(contents.get(contents.size() - 1).key());
|
||||
}
|
||||
}
|
||||
while (objectListing.isTruncated());
|
||||
|
||||
|
||||
Reference in New Issue
Block a user