Search This Blog

Thursday, 7 December 2017

Python Quick Reference - Notes

1.       Python Interactive command prompt is called REPL (Read-Eval-Print-Loop)
2.       Python uses indentation for identifying block of code
3.       To get help use help(modulename) like help(math)
4.       Python is strongly typed and dynamically typed object-oriented programming language
5.       Scalar or primitive types are
a.        Int
                                                               i.      Unlimited precision signed interger
                                                             ii.      By default specified as decimal value like 43
                                                           iii.      Can also be specified as either Binary (using 0b prefix like 0b10) or Octal(using 0o prefix like 0o10) or Hexa(using 0x prefix like 0x10)
b.        float
c.        bollean (true or false)
d.       None (null object)
6.       Type conversion can be done using type function like int(3.5) will result 3
7.       Special floating-point values “nan” not a number, “inf” infinity
8.       If expr:
     Block
Elif
     Block
9.       while expr:
           Block
10.   for a in b[]:
       Block
11.   break breaks entire loop and continue skips current iteration and continues with next iteration
12.   pass does nothing
13.   str (used as string type) is a immutable sequence of Unicode code points
14.   strings are used with quotes (either single or double)..
15.   “”” and “”” is used for multiline comments, also used for multiline strings when used with assignment
16.   # used for commenting lines
17.    THERE IS NO SPEPERATE TYPE FOR SINGLE CHARACTER.
18.   str type provide many string methods like capitalize, split etc
19.   list are mutable sequence of objects.. ex. [‘b’,’c’,’e’,’f’]
20.   dictionaries pair of key: values ex {“India”: “Delhi”, “USA”: “Washington dc”}
21.   two important idioms __name__ and __main__
a.       module is executed once when its first imported
b.       when module is imported using import command, __name__ will be equal to module name (or .py file name)
c.       when module is executed as script using “python modulename.py” then __name__ will be __main__ therefore if we want to execute the code only if module is called as script then put the code in if block with predicate as “__name” == “__main__”
22.   command line arguments
import sys
var=sys.argv[1]
sys.argv[0] is the module or script file name
23.   In python every thing is an object.. for example if a variable “int x=500” defined, “500” is stored in “int” object and “x” will have reference to “int” object.. i.e. if “x” is reassigned with different value then “500” object is not modified instead “x” reference will be changed to point to new object. In case of list objects, lists are mutable therefore same object gets modified
a.       Id() – function returns unique identifier for an object..ex: id(x)
b.       “is” – operator used to check equality of 2 objects.. i.e are both point to same reference?
c.       Python does really have variables.. its has only the named references to an object
d.       “dir()” function returns all the imported modules, functions and different type including special build in types.. for ex: dir(HelloWord) return all the HelloWord.py file components
e.       “type()” returns type of an object
24.   Lambda functions are simple one line functions
a.       For example “def resultd(x):
b.                                        return x*2
c.       Could be written as “resultd=lambda x: x*2”
25.   Collections
a.       “str”
                                                               i.      “abac ass dfsde”.split()=[“abac”, “ass“,“dfsde”]
                                                             ii.      “abc”+”cdf”=abccdf
                                                           iii.      “abc”*2=abcabc
                                                           iv.      Other functions including .. join, capitalize, format, partitioned etc .. refer help(str)
b.       “list” -- heterogeneous mutable sequence
                                                               i.      Elemenets are idexed from zero.. user subscript like list[index] to access specific element
                                                             ii.      Use slicing technique like list[1:2] to access range of elements
                                                           iii.      User -1,-2 ..  indexing to access from last element
                                                           iv.      List.index(“value”) will give element position in list
                                                             v.      “in” and “not in” can be used for membership test
                                                           vi.      “del list[4]” to delete an element by index and “list.remove(“value”) to remove element by its value
                                                          vii.      “list.append[“value”]” or “list.insert(idex ex.2,”value) for inserting values
                                                        viii.      “,“.join(list) will give a string with elements separated by comma
                                                            ix.      “list.sort(key=len)” will sort elements in list using length of element as key
                                                             x.      “list.reverse()” will reverse elements in list
                                                            xi.      sorted(list) will return list of sorted
                                                          xii.      reversed(list) will return iterable object with reversed list
c.       “dict” – unordered mapping from immutable keys and mutable values
                                                               i.      Keys must be unique in dictionary objects
                                                             ii.      If we want to interate only thorugh values then use dictname.values() iterable.. for items dictname.items(), for keys dictname.keys()
d.       “tuple” – heterogeneous immutable sequence
                                                               i.      Similar to list but elements can not be changed and enclosed in parenthesis
                                                             ii.      For single element tuple use trailing comma ex..(abc,)
                                                           iii.      Parenthesis for mini tuples may be omitted, this is useful unpacking references ex.. high-value, low-value=maxmin([1,4,6,7,8]) .. high-value=8, low-value=1.. maxmin is userdefined function
                                                           iv.      “in” and “not in” can be used for membership test
e.       “range”
                                                               i.      range(5) – will give 0,1,2,3,4
                                                             ii.      range(5,10) – will give 5,6,7,8,9
                                                           iii.      range(5,10,2) – will give 5,7,9
                                                           iv.      enumerate() yields (index,value) tuples.. use this instead of range(leg(j)) misuse of range
f.        “set” – unorderd unique mutable set
                                                               i.      “set()” creates empty set
                                                             ii.      Same as list with curley brackets {}
                                                           iii.      When list converted set, duplicates get deleted
                                                           iv.      Mainly userd for functions list “setname.issubset(setname)”, “setname.issuperset(setname)”, “setname.isdisjoint(setname)” etc
26.   Exception handling
a.       Try… Except..Finnaly
b.       Raise without parameter, it re-raises current exception being handled
c.       “except exception: “ is any other exception to handle
27.   Comprehensions
a.       List comprehension example
                                                               i.      Words=”why sometimes I have believed as many as six impossible things before breaskfast”.split()
                                                             ii.      [len(word) for word in word] is the comprehension to calculate all the words lengths and create new list type [3,9,1,4,8,2,4,2,3,10,6,6,9]
b.       List comprehension syntax is [expr(item) for item in iterable]
c.       Set comprehension is same as list comprehension but uses {}
d.       Dict comprehension is {keyexpr:value expr for item in dict}
28.   Iterator
a.       Itr(interable) to get iterator object
b.       List, set etc are iterable objects
c.       Iterator supports functions like next .. next(iterator) will give next element until end and at the end it throws exception StopIteration
29.   Generators
a.       Sequences are evaluated lazily i.e. it will calculate the next value only on demand
b.       All generators are iterators
c.       We can create generators using “yield” in modules instead of return
d.       Generator supports functions like next .. next(generators) will execute statements in module until next yield statement and keeps track of status. At the end  it throws exception StopIteration
e.       Generator comprehension can be created similar to list comprehension but use () instead of []
30.   Standard libraries
a.       “sys”
                                                               i.      For stderr, stdout, stdin, argv
b.       “pprint” pretty print (from pprint import pprint as pp)
c.       “os” import os
                                                               i.      os.path.realpath(p) for file path, os.stat(p).st_size for file size
d.       “glob” import glob
                                                               i.      glob.glob(“*.py”) for searching for all python files
e.        “math”  from math import sqrt .. similarly all math functions
f.        “inertools” from intertools import islice, count
                                                               i.      Islice provides lazy slicing functionality
                                                             ii.      Count() open ended range.. I,e end less range
                                                           iii.      If we need to read all first 1000 primes we can use something like islice(all_primes,1000)… islice((x for x in count() if is_prime(x)),1000)
                                                           iv.      “any” , “all” itertools are similar to logical operators AND and OR but works on interable series and returns bool type true or false
                                                             v.      “zip” combines 2 or more pairs of sequences into matching pair tuples a= [1,2] and b=[a,b] then zip(a,b)=[(1,a),(2,b)].. zip can accept more then 2 interables as input
31.   Classes
a.       “class classname:” for defining a class
b.       “self” first argument to all instant methods in class
c.       Instance methods are functions which can be called on objects
d.       __init__ () is initialization method similar to constructors in Java but not same
e.       There are no access modifiers, everything is public type
f.        Inheritance or extend can be achieved by suppling parent class as parameter to class like “class classname(parentclass):”
g.       Overloading is achieved using duck type not really with different parameters like java
h.       Class variables can be defined outside all functions/methods in a class.. these class variables will be same for all class instances.. this is similar to static variables in Java
i.         “super.” Can be used similar to “self.” But to refer parent class object in the child(or inherited) class
32.   File management
a.       F=Open()
                                                               i.      Input parameters are “file name”,”mode”,”encoding”
                                                             ii.      Mode default is read and text mode
                                                           iii.      Encode default is  UTF-8
                                                           iv.      Open(“file.txt”,”rb”) opens file in read mode and binary mode
                                                             v.      R-read, w-write, a-append, x-exclusive to create new file and fail if file already present
b.       f.write()
                                                               i.      writes stream of data , does not automatically create new lines.. must supply exclusive \n characters
                                                             ii.      file contents are visible only after f.close()
c.       f.read()
                                                               i.      f.read(4) reads 4 bytes
                                                             ii.      accepts number of characters to read and pointer moves to next non-read characters
                                                           iii.      at the end of the file read() returns empty string
d.       f.seek()
                                                               i.      rewind the file position with seek
                                                             ii.      accepts number of position to place the point back in opened file
e.       f.readline() reads a line at a time
f.        f.writeline() writes a line at a time
g.       f.writelines() writes multiple lines at a time
h.       f.close() – closes file
33.   context managers
a.       “with” provides the context management.. i.e. even if we do not close the file explicitly, “with” closes the file at the end of context
b.       with  open(“test.txt”,”w”) as ff
c.       context manager protocol ... from contextLin import closing..
1.       closing(fucntionname whre close needs to be automatically hanled)
34.   UnitTest
a.       Automated and repeatable tests
b.       Key concepts
                                                               i.      TestCase – basic unit of test organization in unittest
                                                             ii.      Fixtures – are features of code that run before and/or after test function
                                                           iii.      Assertions – specific tests for test cases to determine pass or fail ex: is_decimal():
c.       Fixtures avaiables
                                                               i.      “def setUp(self):” is test cases method available to run code before test functions
                                                             ii.      “def tearDown(self):” is test cases method available to run code after test functions
d.       Test functions are defined as testcase mentod with name starting with “test_” ex: “def test_file_present(self):”
35.   PDB – Python DeBugger
a.       This is REPL debugger
b.       “import pdb” then use pdb.set_trace() to start debugger from middle of code
c.       Type help to list all pdb commands
d.       To run a script in debugger mode use “python -m pdb scriptname.py”
e.       In pycharm IDE, we can just put breakpoint and run then step into or step over to do debugging
36.   Virtual environments
a.       Virtual environment is a light weight , self-contained python installation user can create without admin privileges on system
b.       “venv” or “virtualenv” are tools we can use to create virtual environment. “venv” is shipped with python3 if not we can install virtualenv from web
c.       To create new virtual environment simply run “venv” as script as “python -m venv venv-name”  (venv-name is virtual environment directory name)
d.       Once created we can run active scripts from ./bin directory (on linux source it and in window just run it)
e.       Use “deactive” to leave from virtual environment
37.   Packaging and distribution
a.       Using standard “distutils” module
                                                               i.      Create steup.py as
1.       “from distutils.core import setup”
2.       setup(
3.                  name=’packagename’
4.                  version=1.0
5.                  py_modules=[all py modules without .py]
6.                 Any other meta data like author , description etc
7.                  )
8.       Once setup.py created, it can be installed in virtual environment using “python setup.py install”
9.       We can use setup.py to create different formats of distributions for example “python setup.py sdist --format zip” will create zip file for distribution
b.       Use “pip” for thirdparty packages installation. This is similar to “apt-get”,”zipper”,”yum” in linux distributions
c.       We can use “pyinstaller” to create binaries like .exe installable files.. these can be run on any computer without python being installed
d.       To create a set up wizard, we can install tools like “inno setup”

Sunday, 29 October 2017

Linux Package Management



To install a software in Linux servers, Linux uses Packages.  Package contains a file that contains information such as dependent information, version information , architecture information etc.

There are 2 commonly used package systems
1. RPM
2. DEBIAN

The source code or programs that are shipped in .rpm or .deb file will be locally stored and then installed. This is done using RPM or DPKG commands.

Most of distributions now offer online repositories, there is no need to store programs and dependent library package locally before installation. These online repositories configured locally.  Distribution specific tool like APT-GET , YUM or ZYPPER could install/update packages using online repositories.


Tuesday, 26 July 2016

Linux - Identify the disk type and disk partitions where mount points allocated

We can use the command LSSCSI. This will show disk device type and model.

In order to find which physical device its assigned to , we can use command line 
LS -L /sys/block/sda

This will give long string of "../../devices/pci0000:00/0000:00:0d.0/host..." where "0000:00:0d.0" is a PCI device ID. Use "lspci" to identify it. LSPCI is in sbin therefore require superuser access

"parted -l" also gives information about disks.

We can check the file "/proc/partitions" for all list of disk partitions to which mount points are linked.

all the mount points can be seen in the file "/proc/mounts"
"fdisk" could be used to analyse the disks but it requires super user access.

If you are looking for IBM SAN disks , as an option we can check file system as GPFS
Note from Wiki: IBM discontinued selling the SAN File System in April 2007. It has been replaced by IBM General Parallel File System(GPFS).

Monday, 4 July 2016

Working with Memory in Unix

In Unix, files are arranged in file hierarchy, this hierarchy called file system. Files in different devices or on same devices can be mounted to one hierarchy so that all these files can be accessed as single file system. Each mount point can be assigned a space limit. The mount points, space usage and available space on each mount point can be listed using following command.
DF -H (all small letters)

When mount point reached max space limited then processes writing to that mount point will start failing as space is not available for that mount point even though lots of space available in disk.

In this scenario we have two options,
         1)  Either increase space limit for mount point (If more space available in disk)
         2) Purge some files to release space
 
The command that can be used to increase the logical extents for the mount point is lvextend .
The following article explains how to increase space limit on mount point
In terms of releasing the space.. it will be as easy as using “rm -rf”

But if do not want to delete existing folder structure and delete only the old or big files, we need to do following
          1) Identify the folder which occupied more space
          2)   Delete older and larger files

To identify the folders that occupied more space in a mount point, we can use following command
DU -H –MAX-DEPTH=1 (all small letters)

To delete files bases on size and time, use find as below
FIND /TMP/VAR -TYPE F -MTIME +15 -SIZE +2M -EXEC RM -RF {} \; (all small letters)


If we want to check the RAM size and memory usage in linux system, we could use either of following commands

view /proc/meminfo file

or use "free" command

or use vmstat for memory statistics. 



Tuesday, 21 June 2016

Working with Dates in Oracle

Oracle DATE data type could be used to store both date and time parts. Does this mean it always stores both date and time parts or do we have to explicitly store time part? If it automatically stores time part, why time part is not displayed when I query dates in SQL developer or SQL plus? Can I use arithmetic functions directly with date fields? If yes, does the result contains value in days or months or years? How do we extract a specific part (i.e. day or year or secs etc) from date field?
This article is aimed to answer most of these questions.

Storage
Oracle DATE data type by default stores both date and time parts.  
The DATE datatype stores the year (including the century), the month, the day, the hours, the minutes, and the seconds (after midnight).
While inserting data, if time part is ignored then it will store zeros (i.e.  00:00:00 A.M.) in time part.  Remember, DATE data type does not store nano seconds.. if we need fraction of seconds to be stored, then we need to use timestamp.
We could use DUMP function to get the storage details
select dump (sysdate) from dual;

DUMP(SYSDATE)
_________________________________________
Typ=13 Len=8: 7,218,11,26,14,21,24,0

The above result indicates that data is stored as 8 bytes field.

Display
Oracle DATE fields are displayed as per the NLS_DATE_FORMAT settings in oracle database.
select * from SYS.NLS_DATABASE_PARAMETERS where parameter='NLS_DATE_FORMAT';
To display date in different format, change the value for NLS_DATE_FORMAT
Ex. alter session set nls_date_format='DD/MM/YYYY HH.MI.SS';
Or
dbms_session.set_nls('nls_date_format', 'DD/MM/YYYY HH.MI.SS');
For changing at database level, use ALTER SYSTEM.

Date Arithmetic Functions
In order to understand how dates could be used in the arithmetic functions like add, multiple etc.. we need to first understand what data types are accepted on arithmetic functions and also oracle data type internal conversions.
Excerpt from Oracle documentation:
You can use an arithmetic operator with one or two arguments to negate, add, subtract, multiply, and divide numeric values. Some of these operators are also used in datetime and interval arithmetic. The arguments to the operator must resolve to numeric datatypes or to any datatype that can be implicitly converted to a numeric datatype.
Since dates are stored as numbers in the oracle database, we can perform add/subtract arithmetic functions on dates, but multiply/divide are not supported on dates.
When two dates are used in binary +/- operations, the results will be displayed as number of days. If date fields also have time part then result will also have fraction of days (ex 2.33 , 3.44 etc). If we want the result as complete integer value, then we will have to truncate the time part using TRUNC function. The reason for result being number of days (with datatype as integer) is, the dates are stored as days (with data type as number) internally.
Therefore, apart from able to perform +/- on two dates, we can also perform +/- between date and integer (representing number of days).

Important date functions

EXTRACT
  This date function could be used to extract date or time part from fileds with datetime or interval data types
  Ex: select extract(month from sysdate) from dual;

ADD_MONTHS
   This function returns date plus interger months… ex.. add_months(sysdate,1) will return next month date

INTERVAL
   We have both INTERVAL date function and INTERVAL date data type. Both cases, it is a means of dealing with date/time intervals. For example if we want to store year to month interval (ex 2 years 2 months) then we can added column with data type as INTERVAL YEAR [(year_precision)] TO MONTH.
Similarly if we want to just increate the date by 20 seconds then we can use interval function as
sysdate + INTERVAL '20' SECOND
try below query
select to_char(sysdate,'YYYY/MM/DD HH:MI:SS'),to_char(sysdate + INTERVAL '20' SECOND,'YYYY/MM/DD HH:MI:SS') from dual;

MONTHS_BETWEEN
This function returns number of months between two dates
For example select Months_between('02-FEB-15','01-JAN-15') from dual; returns
1.03225806451612903225806451612903225806
The fraction of value return because the difference is 1 month and 1 day .. this one day is converted to month in result.

ROUND and TRUNC
Both round and trunc date functions do same thing.
TRUNC is meant to truncate the time part and ROUND is meant to round the date to nearest day/year or month etc..
For example if the date field has “2016/06/21 07:53:35” then TRUNC function returns 21-JUN-16 but ROUND function return 22-JUN-16.
You can see the difference in these functions if we do not specific the format string and time part in date field is after 12:00 noon. Otherwise both these functions return same result.
We can use ROUND and TRUNC function to return the first day of the year (i.e. using function as TRUNC(DATE,YEAR)) similarly month and week etc..

Wednesday, 25 May 2016

Continuous Integration in software development

In software development process, continuous integration plays very important role. By the way what is continuous integration?
Imagine, we have a unit test, integration test and UAT environments. Every developer could modify everything in unit test environment. But no developer could modify the code or compile the code in higher environments like integration test or UAT. Therefore, any fix or code change tested in unit test environment needs to be elevated or promoted or deployed to higher environments before it could be tested in higher environment.  In order to elevate to next environment, we first check-in code to central repository or version control tool. So the code will be just in one repository that is version controlled. Then the code from version controlled repository will be build (i.e. compiling, linking, binding variables etc) and executables will be deployed to next higher environment or a workspace from where elevation happens. This process is called integration. If we are making changes and check-in code to version controlled repository very frequently and executables are regenerated continuously in next higher environment or a workspace, then it will become continuous integration. Continuous integration normally be automated so that as soon as check-in happens, build will start and executables will be deployed to next environment or workspace or server, from there elevation to next environment takes place.  
Continuous Integration process varies depending on type of code. The above explanation holds good if code is, for example, Java that need to be compiled, JAR or WAR need to be created. But if it is just unix shell scripting which do not require compiling, the continuous integration may be just copying the code from version control tool to designated environment therefore we may not see any continuous integration tools like Hudson or Jenkins playing any role in Unix Shell Scripting projects as it could be automated with simple scripting in native OS.


Assuming we are working in a JAVA project, the continuous integration could be implanted as below