org.apache.hadoop.hbase.test

Class IntegrationTestLoadAndVerify

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

IntegrationTestWithCellVisibilityLoadAndVerify

public class IntegrationTestLoadAndVerify
extends IntegrationTestBase

A large test which loads a lot of data that has internal references, and verifies the data. In load step, 200 map tasks are launched, which in turn write loadmapper.num_to_write (default 100K) rows to an hbase table. Rows are written in blocks, for a total of 100 blocks. Each row in a block, contains loadmapper.backrefs (default 50) references to random rows in the prev block. Verify step is scans the table, and verifies that for every referenced row, the row is actually there (no data loss). Failed rows are output from reduce to be saved in the job output dir in hdfs and inspected later. This class can be run as a unit test, as an integration test, or from the command line Originally taken from Apache Bigtop.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestLoadAndVerify.LoadMapper
static class	IntegrationTestLoadAndVerify.VerifyMapper
static class	IntegrationTestLoadAndVerify.VerifyReducer
static class	IntegrationTestLoadAndVerify.WALSearcher Tool to search missing rows in WALs and hfiles.

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestLoadAndVerify()

Method Summary

Modifier and Type	Method and Description
void	cleanUpCluster()
protected org.apache.hadoop.mapreduce.Job	doLoad(org.apache.hadoop.conf.Configuration conf, HTableDescriptor htd)
protected void	doVerify(org.apache.hadoop.conf.Configuration conf, HTableDescriptor htd)
protected Set<String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
org.apache.hadoop.fs.Path	getTestDir(String testName, String subdir)
static void	main(String[] argv)
void	printUsage()
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)
int	runTestFromCommandLine()
protected void	setMapperClass(org.apache.hadoop.mapreduce.Job job)
void	setUpCluster()
static long	swapLong(long value) Converts a "long" value between endian systems.
void	testLoadAndVerify()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

addOptions, cleanUp, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestLoadAndVerify

public IntegrationTestLoadAndVerify()

Method Detail

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

cleanUpCluster

public void cleanUpCluster()
                    throws Exception

Overrides:

cleanUpCluster in class IntegrationTestBase

Throws:

Exception

swapLong

public static long swapLong(long value)

Converts a "long" value between endian systems. Borrowed from Apache Commons IO

Parameters:

value - value to convert

Returns:

the converted value

doLoad

protected org.apache.hadoop.mapreduce.Job doLoad(org.apache.hadoop.conf.Configuration conf,
                                                 HTableDescriptor htd)
                                          throws Exception

Throws:

Exception

setMapperClass

protected void setMapperClass(org.apache.hadoop.mapreduce.Job job)

doVerify

protected void doVerify(org.apache.hadoop.conf.Configuration conf,
                        HTableDescriptor htd)
                 throws Exception

Throws:

Exception

getTestDir

public org.apache.hadoop.fs.Path getTestDir(String testName,
                                            String subdir)
                                     throws IOException

Throws:

IOException

testLoadAndVerify

public void testLoadAndVerify()
                       throws Exception

Throws:

Exception

printUsage

public void printUsage()

Overrides:

printUsage in class AbstractHBaseTool

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] argv)
                 throws Exception

Throws:

Exception