Abinitio Performance

Ab Initio PERFORMANCE


How To Improve Performance: -


1. Go Parallel as soon as possible using Ab Initio Partitioning technique.
2. Once Data Is partitioned do not bring to serial , then back to parallel. Repartition instead.
3. For Small processing jobs serial may be better than parallel.
4. Do not access large files across NFS, Use FTP component
5. Use Ad Hoc MFS to read many serial files in parallel and use concat coponenet.
· Don’t use filter by Expression. Most of the components has embedded filter by expression called select expression use embedded select instead of Filter by expression if possible to improve performance.




CONCATENATE
8






Ad Hoc MFS(80 files)


1. Using Phase breaks let you allocate more memory to individual component and make your graph run faster
2. Use Checkpoint after the sort than land data on to disk
3. Use Join and rollup in-memory feature
4. Best performance will be gained when components can work with in memory by MAX-CORE.
5. MAR-CORE for SORT is calculated by finding size of input data file.
6. For In-memory join memory needed is equal to non-driving data size + overhead.
7. If in-memory join cannot fir its non-driving inputs in the provided MAX-CORE then it will drop all the inputs to disk and in-memory does not make sence.
8. Use rollup and Filter by EX as soon as possible to reduce number of records.
9. When joining very small dataset to a very large dataset, it is more efficient to broadcast the small dataset to MFS using broadcast component or use the small file as lookup.

Reduce number of components may save startup costs.
Don’t use MFS if you have small datasets
Use select filter inside the component than separate Filter By Ex component
6. Monitor UNIX CPU usage by using vmstat , disk usage using iostat .