#6 Support for pieced downloads greater than 2^31 -1 bytes

[Rodney Beede]

Support for pieced downloads greater than 2^31 -1 bytes is not possible because byte arrays that are indexed on integers are used. The integer has a maximum value of 2^31 - 1 bytes which can cause this limitation.

Suppose a file of size 8,589,934,592 (4 * 2^31) were to be downloaded. When divided into 4 pieces each piece has to download 2^31 bytes which cannot be stored in an array that is indexed by int.

Also other Java Collections like List use int indexing which can't support that many bytes. Granted the machine would have to be 64-bit and have lots of memory, but as of Java 6 there isn't support for indexing with anything more than int.

One possible work around would be to use multi-dimensional arrays. Either way it will require a custom implementation unless Java adds support for indexing by long or some other data type.

Another possible solution is to devise a way to write large amounts of data to a temporary file on disk. This may be slower in terms of concatenating all the pieces together though. Ideally when a File is passed in each piece writes to its isolated section of the file directly, and when only a OutputStream is given then temporary files are created for streaming back into the single stream.

[Rodney Beede]

Dynamically adjusting the number of pieces should resolve this issue. It would still allow the code to easily stream the data into manageable chunks.