I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.

It’s a tuple of dicts of tuple of array.array.

Can pickle dump in chunks? If not, is there another technique I can use?

  • radarsat1@lemmy.ml
    link
    fedilink
    arrow-up
    3
    ·
    1 year ago

    If you’re pickling that much data you should definitely consider using a more appropriate data format. Maybe a database or HDF5?

    • lntl@lemmy.mlOP
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Agreed. When I started, things were much simpler. Trying not to revise too much code but I can if there’s no other option.