IBM Infosphere DataStage - Hadoop Connectivity Configuration

BigData File Stage or BDFS stage was introduced as part of IBM Infosphere DataStage V8.7

DataStage8.7NewFeatures

In Version 9.1, Balanced Optimization features of InfoSphere DataStage are extended to Hadoop BDFS also. I.e. from V9.1 onwards, we can push sets of data integration processing and related data I/O into a Hadoop cluster.

 

In Version 11.3, the Big Data File stage compatibility is extended to Hortonworks 2.1, Cloudera 4.5, and InfoSphere BigInsights 3.0. 

 

 

In Version 11.5, all features to assess and monitor the quality of your data are now supported executing directly inside a Hadoop cluster. You can run the same profiling and data rules inside or outside your Hadoop environment,

 

 

Before using BDFS Stage, we need configure the Hadoop connectivity as below

•  The shared library libhdfs.so should have correct level of access so that BDFS stage can access this library


• For the correct functioning of the libhdfs.so library, we need to add the HDFS .jar files path to CLASSPATH variables


• add HDFS configuration file hdfs-site.xml  path to CLASSPATH variable


• add the directory that contains the libhdfs.so file to the LD_LIBRARY_PATH variable


• Add the JDK bin directory to the PATH variable


• add fs.hdfs.impl property in the core-site.xml file if this property not present.


• <property>
      <name>fs.hdfs.impl</name>
      <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
  </property>


• Set environment variable HADOOP_CONF_DIR pointing to Hadoop configuration files.