TFile, SequenceFile与gz,lzo压缩的测试

先记一记,以后解释 :)

 

$hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison -c gz -f seqfile -r /home/zhoumin/tmp -x rw -b 65536 -s 1024
=== SeqFile: Creation (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
10/01/07 22:31:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/01/07 22:31:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
10/01/07 22:31:35 INFO compress.CodecPool: Got brand-new compressor
2010-01-07 22:31:35  Start writing: SeqFile.Performance...
2010-01-07 22:31:35 Started timing.
2010-01-07 22:34:19 Stopped timing.
2010-01-07 22:34:19  Duration: 164035ms...total size: 3078.62MB...raw thrpt: 18.77MB/s
2010-01-07 22:34:19  Compressed size: 1024.29MB...compressed thrpt: 6.24MB/s.
=== SeqFile: Reading (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
10/01/07 22:34:19 INFO compress.CodecPool: Got brand-new decompressor
10/01/07 22:34:19 INFO compress.CodecPool: Got brand-new decompressor
10/01/07 22:34:19 INFO compress.CodecPool: Got brand-new decompressor
10/01/07 22:34:19 INFO compress.CodecPool: Got brand-new decompressor
2010-01-07 22:34:19  Start reading: SeqFile.Performance...
2010-01-07 22:34:19 Started timing.
2010-01-07 22:34:56 Stopped timing.
2010-01-07 22:34:56  Duration: 37222ms...total size: 3078.62MB...raw thrpt: 82.71MB/s
2010-01-07 22:34:56  Compressed size: 1024.29MB...compressed thrpt: 27.52MB/s.

$hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison -c gz -f tfile -r /home/zhoumin/tmp -x rw -b 65536 -s 1024
=== TFile: Creation (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
2010-01-07 22:28:13  Start writing: TFile.Performance...
2010-01-07 22:28:13 Started timing.
10/01/07 22:28:13 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/01/07 22:28:13 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
10/01/07 22:28:13 INFO compress.CodecPool: Got brand-new compressor
2010-01-07 22:30:50 Stopped timing.
2010-01-07 22:30:50  Duration: 157829ms...total size: 3484.39MB...raw thrpt: 22.08MB/s
2010-01-07 22:30:50  Compressed size: 1024.05MB...compressed thrpt: 6.49MB/s.
=== TFile: Reading (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
10/01/07 22:30:50 INFO compress.CodecPool: Got brand-new decompressor
2010-01-07 22:30:50  Start reading: TFile.Performance...
2010-01-07 22:30:50 Started timing.
2010-01-07 22:31:22 Stopped timing.
2010-01-07 22:31:22  Duration: 31441ms...total size: 3484.39MB...raw thrpt: 110.82MB/s
2010-01-07 22:31:22  Compressed size: 1024.05MB...compressed thrpt: 32.57MB/s.


$hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison -c lzo -f tfile -r /home/zhoumin/tmp -x rw -b 65536 -s 1024    
=== TFile: Creation (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...lzo) ===
2010-01-07 22:35:21  Start writing: TFile.Performance...
2010-01-07 22:35:21 Started timing.
10/01/07 22:35:21 INFO tfile.Compression: Trying to load Lzo codec class: org.apache.hadoop.io.compress.LzoCodec
10/01/07 22:35:21 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/01/07 22:35:21 INFO compress.LzoCodec: Successfully loaded & initialized native-lzo library
10/01/07 22:35:21 INFO compress.CodecPool: Got brand-new compressor
2010-01-07 22:36:13 Stopped timing.
2010-01-07 22:36:13  Duration: 52223ms...total size: 2108.38MB...raw thrpt: 40.37MB/s
2010-01-07 22:36:13  Compressed size: 1024.08MB...compressed thrpt: 19.61MB/s.
=== TFile: Reading (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 65536.00KB...Target Size: 1024.00MB...Compression: ...lzo) ===
10/01/07 22:36:14 INFO compress.CodecPool: Got brand-new decompressor
2010-01-07 22:36:14  Start reading: TFile.Performance...
2010-01-07 22:36:14 Started timing.
2010-01-07 22:36:28 Stopped timing.
2010-01-07 22:36:28  Duration: 14685ms...total size: 2108.38MB...raw thrpt: 143.57MB/s
2010-01-07 22:36:28  Compressed size: 1024.08MB...compressed thrpt: 69.74MB/s.

 

 

hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison -c gz -f seqfile -r /home/zhoumin/tmp -x rw -b 2048 -s 1024       
=== SeqFile: Creation (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 2048.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
10/03/19 00:04:05 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/03/19 00:04:05 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
10/03/19 00:04:05 INFO compress.CodecPool: Got brand-new compressor
2010-03-19 00:04:05  Start writing: SeqFile.Performance...
2010-03-19 00:04:05 Started timing.
2010-03-19 00:06:53 Stopped timing.
2010-03-19 00:06:53  Duration: 168253ms...total size: 3080.42MB...raw thrpt: 18.31MB/s
2010-03-19 00:06:53  Compressed size: 1024.23MB...compressed thrpt: 6.09MB/s.
=== SeqFile: Reading (KLEN: 50-100... VLEN: 100-200...MinBlkSize: 2048.00KB...Target Size: 1024.00MB...Compression: ...gz) ===
10/03/19 00:06:53 INFO compress.CodecPool: Got brand-new decompressor
10/03/19 00:06:53 INFO compress.CodecPool: Got brand-new decompressor
10/03/19 00:06:53 INFO compress.CodecPool: Got brand-new decompressor
10/03/19 00:06:53 INFO compress.CodecPool: Got brand-new decompressor
2010-03-19 00:06:53  Start reading: SeqFile.Performance...
2010-03-19 00:06:53 Started timing.
2010-03-19 00:07:31 Stopped timing.
2010-03-19 00:07:31  Duration: 37375ms...total size: 3080.42MB...raw thrpt: 82.42MB/s
2010-03-19 00:07:31  Compressed size: 1024.23MB...compressed thrpt: 27.40MB/s.

 

 

 

hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeek -c gz -f seqfile -r /home/zhoumin/tmp -x rw -b 1024 -s 2048    
10/03/18 21:40:09 INFO tfile.Compression: Trying to load Lzo codec class: org.apache.hadoop.io.compress.LzoCodec
10/03/18 21:40:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/03/18 21:40:09 INFO compress.LzoCodec: Successfully loaded & initialized native-lzo library
10/03/18 21:40:09 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
10/03/18 21:40:09 INFO compress.CodecPool: Got brand-new compressor
time: 347.10s...uncompressed: 6430.03MB...raw thrpt: 19.42MB/s
time: 347.10s...file size: 2048.07MB...disk thrpt: 6.19MB/s
10/03/18 21:45:56 INFO compress.CodecPool: Got brand-new decompressor
time: 17.28s...avg seek: 17.28ms...1000 hit...0 miss...avg I/O size: 0.16KB

 

 

hadoop jar tfile-0.00.1-dev.jar org.apache.hadoop.io.file.tfile.TestTFileSeek -c gz -f seqfile -r /home/zhoumin/tmp -x rw -b 2048 -s 2048
10/03/18 21:32:09 INFO tfile.Compression: Trying to load Lzo codec class: org.apache.hadoop.io.compress.LzoCodec
10/03/18 21:32:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
10/03/18 21:32:09 INFO compress.LzoCodec: Successfully loaded & initialized native-lzo library
10/03/18 21:32:09 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
10/03/18 21:32:09 INFO compress.CodecPool: Got brand-new compressor
time: 352.27s...uncompressed: 6464.91MB...raw thrpt: 19.24MB/s
time: 352.27s...file size: 2048.09MB...disk thrpt: 6.10MB/s
10/03/18 21:38:02 INFO compress.CodecPool: Got brand-new decompressor
time: 33.07s...avg seek: 33.07ms...1000 hit...0 miss...avg I/O size: 0.16KB