org.apache.hadoop.hbase.mapreduce

Class IntegrationTestBulkLoad

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class IntegrationTestBulkLoad
extends IntegrationTestBase

Test Bulk Load and MR on a distributed cluster. It starts an MR job that creates linked chains The format of rows is like this: Row Key -> Long L:<< Chain Id >> -> Row Key of the next link in the chain S:<< Chain Id >> -> The step in the chain that his link is. D:<< Chain Id >> -> Random Data. All chains start on row 0. All rk's are > 0. After creating the linked lists they are walked over using a TableMapper based Mapreduce Job. There are a few options exposed: hbase.IntegrationTestBulkLoad.chainLength The number of rows that will be part of each and every chain. hbase.IntegrationTestBulkLoad.numMaps The number of mappers that will be run. Each mapper creates on linked list chain. hbase.IntegrationTestBulkLoad.numImportRounds How many jobs will be run to create linked lists. hbase.IntegrationTestBulkLoad.tableName The name of the table. hbase.IntegrationTestBulkLoad.replicaCount How many region replicas to configure for the table under test.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestBulkLoad.CompositeKeyComparator Comparator used to order linkKeys so that they are passed to a reducer in order.
static class	IntegrationTestBulkLoad.EmptySplit
static class	IntegrationTestBulkLoad.FixedRecordReader<K,V>
static class	IntegrationTestBulkLoad.ITBulkLoadInputFormat
static class	IntegrationTestBulkLoad.LinkChain Writable used as the value emitted from a pass over the hbase table.
static class	IntegrationTestBulkLoad.LinkedListCheckingMapper Mapper to pass over the table.
static class	IntegrationTestBulkLoad.LinkedListCheckingReducer Class that does the actual checking of the links.
static class	IntegrationTestBulkLoad.LinkedListCreationMapper Mapper that creates a linked list of KeyValues.
static class	IntegrationTestBulkLoad.LinkKey Writable class used as the key to group links in the linked list.
static class	IntegrationTestBulkLoad.NaturalKeyGroupingComparator Comparator used to figure out if a linkKey should be grouped together.
static class	IntegrationTestBulkLoad.NaturalKeyPartitioner Class to figure out what partition to send a link in the chain to.
static class	IntegrationTestBulkLoad.SlowMeCoproScanOperations

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestBulkLoad()

Method Summary

Modifier and Type	Method and Description
protected void	addOptions()
protected Set<String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
static TableName	getTableName(org.apache.hadoop.conf.Configuration conf)
static void	main(String[] args)
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)
void	runLoad()
int	runTestFromCommandLine()
void	setUpCluster()
void	testBulkLoad()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestBulkLoad

public IntegrationTestBulkLoad()

Method Detail

testBulkLoad

public void testBulkLoad()
                  throws Exception

Throws:

Exception

runLoad

public void runLoad()
             throws Exception

Throws:

Exception

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

addOptions

protected void addOptions()

Overrides:

addOptions in class IntegrationTestBase

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getTableName

public static TableName getTableName(org.apache.hadoop.conf.Configuration conf)

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception