create_stream_for_chunk builds up 8mb chunk of memory

[silverjam]

In create_stream_for_chunk the stated purpose is to create a stream of small chunks of memory read from a Reader object however it appears that during the aws_sdk migration there was a need to transform the stream to a different type per the comment here:

esthri/crates/esthri/src/ops/upload.rs

Line 367 in 7cbb824

while let Some(bytes) = stream.try_next().await? {

This while loop ends up building up an 8mb chunk of memory anyway. If this is an issue for memory usage in the future I would suggest remove this function and just passing a whole 8mb chunk to the multi-part PUT call, or fix the method (somehow) so that the streaming behavior can be maintained.

[pcrumley]

Thanks for pointing this out. It seems like a somewhat minor issue, unless I am missing something.

I'll need to see if we can remove the method or if it needs to be fixed.