Class ShardedFile

  • All Implemented Interfaces:
    java.io.Serializable

    public final class ShardedFile
    extends java.lang.Object
    implements java.io.Serializable
    See Also:
    Serialized Form
    • Field Summary

      Fields 
      Modifier and Type Field Description
      private java.io.File[] myShards  
      int numberOfShards  
      private static long serialVersionUID  
      java.io.File single
      A valid single file used as a template when creating the shards.
    • Constructor Summary

      Constructors 
      Constructor Description
      ShardedFile​(java.io.File template, int nbShards)  
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void delete()
      Explicitly delete all files/shards as well as the parent directory (make sure there is nothing else in that directory)
      java.io.File directory()  
      boolean equals​(java.lang.Object obj)  
      java.util.List<java.io.File> files()  
      java.util.List<EntryPair.KeyedPrimitive<java.io.File>> filesWithShardIndex()  
      int hashCode()  
      <T> java.util.function.Supplier<FromFileReader<T>> newSequencedFactory​(java.util.function.Function<java.io.File,​FromFileReader<T>> factory)
      Each reader instantiated by this factory will read from the shards in sequence, until all of them are done.
      static ShardedFile of​(java.io.File template, int nbShards)  
      static ShardedFile of​(java.io.File templateFolder, java.lang.String templateFile, int nbShards)  
      java.io.File shard​(int index)  
      java.io.File[] shards()  
      private static java.io.File[] splitToShards​(java.io.File file, int numberOfShards)  
      java.lang.String toString()  
      • Methods inherited from class java.lang.Object

        clone, finalize, getClass, notify, notifyAll, wait, wait, wait
    • Field Detail

      • numberOfShards

        public final int numberOfShards
      • single

        public final java.io.File single
        A valid single file used as a template when creating the shards. Can also be used for meta data or merged/aggregated data from the shards.
      • myShards

        private transient java.io.File[] myShards
    • Constructor Detail

      • ShardedFile

        ShardedFile​(java.io.File template,
                    int nbShards)
    • Method Detail

      • of

        public static ShardedFile of​(java.io.File template,
                                     int nbShards)
      • of

        public static ShardedFile of​(java.io.File templateFolder,
                                     java.lang.String templateFile,
                                     int nbShards)
      • splitToShards

        private static java.io.File[] splitToShards​(java.io.File file,
                                                    int numberOfShards)
      • delete

        public void delete()
        Explicitly delete all files/shards as well as the parent directory (make sure there is nothing else in that directory)
      • directory

        public java.io.File directory()
        Returns:
        A parent directory to all the shards
      • equals

        public boolean equals​(java.lang.Object obj)
        Overrides:
        equals in class java.lang.Object
      • files

        public java.util.List<java.io.File> files()
        Returns:
        Same as shards() but as a List.
      • filesWithShardIndex

        public java.util.List<EntryPair.KeyedPrimitive<java.io.File>> filesWithShardIndex()
        Returns:
        Same as files() but paired with the shard index.
      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class java.lang.Object
      • newSequencedFactory

        public <T> java.util.function.Supplier<FromFileReader<T>> newSequencedFactory​(java.util.function.Function<java.io.File,​FromFileReader<T>> factory)
        Each reader instantiated by this factory will read from the shards in sequence, until all of them are done. The idea is that you can create multiple readers and have them work in parallel (each shard will only be read once by one of the readers).
      • shard

        public java.io.File shard​(int index)
      • shards

        public java.io.File[] shards()
      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object