org.apache.hadoop.hbase.spark

Class IntegrationTestSparkBulkLoad

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class IntegrationTestSparkBulkLoad
extends IntegrationTestBase

Test Bulk Load and Spark on a distributed cluster. It starts an Spark job that creates linked chains. This test mimic IntegrationTestBulkLoad in mapreduce. Usage on cluster: First add hbase related jars and hbase-spark.jar into spark classpath. spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-spark-it-XXX-tests.jar -m slowDeterministic -Dhbase.spark.bulkload.chainlength=300

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestSparkBulkLoad.createCombinerFunc
static class	IntegrationTestSparkBulkLoad.LinkedListCheckingFlatMapFunc PairFlatMapFunction used to transfer to Tuple
static class	IntegrationTestSparkBulkLoad.LinkedListCheckingForeachFunc Sort all LinkChain for one LinkKey, and test List
static class	IntegrationTestSparkBulkLoad.LinkedListCreationMapper
static class	IntegrationTestSparkBulkLoad.ListToKeyValueFunc
static class	IntegrationTestSparkBulkLoad.mergeCombinersFunc
static class	IntegrationTestSparkBulkLoad.mergeValueFunc
static class	IntegrationTestSparkBulkLoad.NaturalKeyPartitioner Class to figure out what partition to send a link in the chain to.
static class	IntegrationTestSparkBulkLoad.SparkLinkChain Writable used as the value emitted from a pass over the hbase table.
static class	IntegrationTestSparkBulkLoad.SparkLinkKey Writable class used as the key to group links in the linked list.

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestSparkBulkLoad()

Method Summary

Modifier and Type	Method and Description
protected void	addOptions()
protected Set<String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
static TableName	getTableName(org.apache.hadoop.conf.Configuration conf)
static void	main(String[] args)
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)
void	runCheck() After adding data to the table start a mr job to
void	runLinkedListSparkJob(int iteration) Running spark job to create LinkedList for testing
void	runLoad() Running spark job to load data into hbase table
int	runTestFromCommandLine()
void	setUpCluster()
void	testBulkLoad()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestSparkBulkLoad

public IntegrationTestSparkBulkLoad()

Method Detail

runLoad

public void runLoad()
             throws Exception

Running spark job to load data into hbase table

Throws:

Exception

runLinkedListSparkJob

public void runLinkedListSparkJob(int iteration)
                           throws Exception

Running spark job to create LinkedList for testing

Parameters:

iteration - iteration th of this job

Throws:

Exception

runCheck

public void runCheck()
              throws Exception

After adding data to the table start a mr job to

Throws:

IOException

ClassNotFoundException

InterruptedException

Exception

testBulkLoad

public void testBulkLoad()
                  throws Exception

Throws:

Exception

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

addOptions

protected void addOptions()

Overrides:

addOptions in class IntegrationTestBase

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getTableName

public static TableName getTableName(org.apache.hadoop.conf.Configuration conf)

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception