How to partition address space into more manageable sizes?

[bsalmilo]

Hi,

I'm having an issue with the size of the address space that I am retrieving. In my system, after address space retrieval, I compress and upload the Address Space to AWS S3. However, S3 has a retrieval limit of 6MB, and in many cases my Address Space graphs are larger than this. Are there any known ways of partitioning or paginating the Address Space data so that it is made up of smaller pieces, without disrupting the shape of the data?

Thank you

[kevinherron]

What format are you storing these in now? How many Nodes and References are these larger graphs comprised of?

[kevinherron]

Also, S3 does not have a 6mb limit… are you talking about a Lambda?

[bsalmilo]

Yes, I'm trying to retrieve address space in a lambda via getObjectRequest. The AddressSpace object looks like the following

data class AddressSpace(val addressSpace: Map<NodeId, Node>, val namespaceList: List<String> = emptyList())

I was hoping to partition the upload so that I could paginate the retrieval.

[kevinherron]

You're inventing this storage format, partition it however you like. Split the map up for storage in multiple parts, combine it again after reading back all the parts.