Search This Blog
Showing posts with label DataStage. Show all posts
Showing posts with label DataStage. Show all posts

Thursday, 7 April 2016

Some important URLs for technical library

Posted by at No comments:
Labels:

Wednesday, 6 April 2016

IBM Infosphere QualityStage simplified


IBM Infosphere server quality stage is part of information server suit and its available with additional license.  Buying QualityStage will enable us to use more stages (plus match specification and rule designer, for use in these stages).. that’s it.. Everything else is same and part of DataStage..
Do not confuse with Information Analyzer and Data Rules stage is not part of QualityStage, its part of Information Analyzer.

What are the additional stages comes with QualityStage

1.      Investigate

2.      Standardize

3.      One-source Match

4.      Two-source Match

5.      Match Frequency

6.      Survive

7.      SQA (Standardization Quality Assessment)

8.      MNS (Multinational Standardization)

 

If we understand steps in the data quality assurance process, then understanding these stages will be easy. Data quality process steps and corresponding QualityStage Stages are


While working on any DataQuality stages, its important to understand single domain column and free form columns

Single Domain columns represent one specific business attribute… for example first name or customer id…

Free Form columns contains free text, combination of many attributes.. for example name, it could contain first name and last name or first, last and middle name… similarly address, it could contain only the house number and street name or entire address along with post code.

Analyzing single domain columns is easy because we upfront know what data the field contains. But analyzing free form fields is not that easy because we need to first set some rules expecting what free form field might contains..  i.e. to analyses the free form fields, we need to define set of rules (using rule designer).  The rules work on the fact that free form field is nothing but bunch of tokens and then we map each token to a pattern, then decide output action when token is as expected (i.e. conformed to the pattern)

To standardise single domain columns, we may need rule set in some cases for example if we want to lookup correct value or spell correct or change description to acronym or we could just use modify, transformer stage to deal with nulls or change the format etc...  

DataQuality Standardise stage normally used to standardise the free form column to split the free form columns into single domain columns. We need rule sets for this purpose.

Following quality stage use the Rule sets

·        Investigate

·        Standardize

·        Multinational Standardize (MNS)

Rule set contains

·        Classifications

·        Output Columns

·        Rules

o   Rules basically define the input patterns and the actions when pattern matches. As part of this pattern matching we could use pattern specification, classifications tables, look up tables

o   We can specify what action to be taken when pattern matches and what output to be written to output columns

 

When it comes to matching process,

        Matching is nothing but comparing two records and checking if both records are duplicates are not. Then we can use survive stage to keep one record and drop other record in the duplicate pair.

Match stages take match specification as input and match specification could be created using match specification designer.

Match specification tells,

·        On which keys we need to match

·        What are group by columns in order to divide total records into blocks to operate on blocks first instead of operating on entire data set. Dividing entire data into blocks in very important. Because we are not trying to identify only 100% matched records. Two records match by 20% may be good enough for us to decide these records may be duplicates and need to be reviewed by someone before confirming these are duplicates. So we say 20% match are clerical match type. Therefore in order to identify which one is matching which, we need to first divide data into blocks where we are likely to find the duplicates

·        How many passes we need to run the match process before finalizing the match weightage?

More details about quality stage could be found at below links

http://www.ibm.com/developerworks/data/library/techarticle/dm-1505duplicates-infosphere-qualitystage/index.html

http://bigpaathshala.blogspot.com/2014/10/investigate-stage-overview.html

 
Posted by at 3 comments:
Labels:

Tuesday, 5 April 2016

IBM Infosphere datastage - Command Line interface commands for administration


Common administering or administration tasks performed by datastage admin

1.      Managing the assets and asset migration (example exporting security assets example users and user groups)

a.      ISTOOL  à

istool is a command line interface. This is present in both client and engine tiers. The location if CLI is /opt/IBM/InformationServer/Clients/istools/cli

Istool is used for managing assets, i.e. build package, send or deploy package, import/export assets etc

Example:

Istool export -security '-securityUser -userident "*" -includeRoles'

 -domain host:port -username user

 -password password -archive ExportISFSecurityRoles.isx

 

2.      Enable or disable product features or set/unset the product configuration properties

a.      IISADMIN à

Use iisadmin can be used to activate or deactivate an edition or feature pack. More details about iisadmin could found in below link

https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.5.0/com.ibm.swg.im.iis.found.admin.common.doc/topics/r_iisadmincommand.html

 

for example, if we want to see what features are active then we could use below command

./iisAdmin.sh -display -key com.ibm.iis.datastage.license.*

Results..

com.ibm.iis.datastage.license.option.parallel=1

com.ibm.iis.datastage.license.option.qualitystage=0

 

above results indicate that qualitystage is not enabled.

 

3.      Administering services tier i.e. starting or stopping services

a.      Starting services

                                                    i.     First start the application server

1.      Go to /opt/IBM/InformationServer/ASBServer/bin

2.      Run “./MetadataServer.sh start” or “./MetadataServer.sh run”

3.      run echos the output (i.e. runs in forground) and start runs in background

                                                   ii.     Then start engine service

1.      Go to /opt/IBM/InformationServer/Server/DSEngine

2.      ./bin/uv -admin -start

                                                  iii.     Then start ASB Agent

1.      Go to /opt/IBM/InformationServer/ASBNode/bin

2.      ./NodeAgents.sh start

b.      Shutting down or stopping services

                                                    i.     First stop the datastage engine services (Metadata Server services, DSRPC Server etc)

1.      Go to /opt/IBM/InformationServer/Server/DSEngine

2.      ./bin/uv -admin –stop

                                                   ii.     Then stop agents

1.      Go to /opt/IBM/InformationServer/ASBNode/bin

2.      ./NodeAgents.sh stop

                                                  iii.     Then stop application server

1.      Go to /opt/IBM/InformationServer/ASBServer/bin

2.      ./MetadataServer.sh stop

 

4.      AppServerAdmin Command

a.      Run this command whenever admin user account password is changed so that new password is reflected across all the Information Server suite components configuration

5.      SessionAdmin command

a.      Use this to manage and monitor the active sessions

6.      DirectoryAdmin tool

a.      Use this to access metadata repository and user registry, and complete variety of actions on user registry including adding new user, changing password, deleting user/group, changing the user role etc..

7.      DirectoryCommand tool

a.      Similar to directoryadmin but not same. This can be used to add/delete users/groups etc

8.      Encrypt

a.      Use this command to encrypt the user credentials

9.      Orchadmin

a.      Use this command to research on dataset.
ORCHADMIN CHECK  -- Check configuration file

                         COPY, DELETE, DESCRIBE and TRUNCATE datasets

                         DUMP dataset data to readable format files

10.   What is ASB agent

a.      This is the back group process that conveys the requests from client tier to service tier.  

11.   What is dsrpcd (DSRPC Service)

a.      This service allows client tier to connect to server engine.

12.   Another very important command is “dsjob”, though administrator do not normally use this command, its very important to know and understand how dsjob works.
Posted by at No comments:
Labels:

Saturday, 2 April 2016

IBM Infosphere Datastage – Change Data Capture, Difference and Compare stages




 
 

Table 1.
bcol0
bcol1
bcol2
bcol3
bcol4
0
0
0
0
a
1
7
1
1
b
2
2
2
2
c
3
3
3
3
d
4
5
4
4
e
5
2
5
5
f
6
6
6
6
g
7
7
7
7
h
8
8
8
8
i
9
9
9
9
j

 

 

Table 2.
bcol0
bcol1
bcol2
bcol3
bcol4
0
0
0
0
a
1
1
1
1
b
2
2
2
2
C
3
3
3
3
D
4
4
4
4
E
5
5
5
5
F
6
6
6
6
G
7
7
7
7
H
8
8
8
8
I
9
9
9
9
J

This is the data set output by the Change Capture stage (bcol4 is the key column, bcol1 the value column):

Table 3. Change data set
bcol0
bcol1
bcol2
bcol3
bcol4
change_code
1
1
1
1
B
3
4
4
4
4
E
3
5
5
5
5
F
3

 

 

This is the data set output by the Difference stage (Key is the key column, All non-key columns are values are set True, all other settings take the default):

Table 3. Output data set
acol0
acol1
acol2
acol3
key
diff
0
0
0
0
a
2
1
1
1
1
b
3
2
2
2
2
c
2
3
3
3
3
d
2
4
4
4
4
e
3
5
5
5
5
f
3
6
6
6
6
g
2
7
7
7
7
h
2
8
8
8
8
i
2
9
9
9
9
j
2

 

The stage compares on the Key columns bcol1 and bcol4. This is the output data set by compare stage:

Table 3. Output data set
Result
First
Second
 
bcol0
bcol1
bcol2
bcol3
bcol4
bcol0
bcol1
bcol2
bcol3
bcol4
0
0
0
0
0
a
0
0
0
0
a
2
1
7
1
1
b
1
1
1
1
b
0
2
2
2
2
c
2
2
2
2
c
0
3
3
3
3
d
3
3
3
3
d
2
4
5
4
4
e
4
4
4
4
e
-1
5
2
5
5
f
5
5
5
5
f
0
6
6
6
6
g
6
6
6
6
g
0
7
7
7
7
h
7
7
7
7
h
0
8
8
8
8
i
8
8
8
8
i
0
9
9
9
9
j
9
9
9
9
j

 

The operation code by Change data capture Stage are

Copy – 0  (By default copy i.e. no change, records are not output by change data capture stage)

Insert – 1

Deleted – 2

Edit – 3

The operation code by Difference Stage are

Copy – 0  

Insert – 1

Deleted – 2

Edit – 3

The operation code by Compare Stage are

Equals – 0

First is empty – -2

Greater Than – 1

Less than -- -1

Second is empty – 2

 
  
Posted by at 1 comment:
Labels:
Subscribe to: Posts (Atom)