Near Zero Downtime Upgrade

Here's a brief steps for accomplishing a near zero downtime upgrade.


1) A golden client is built with ERP 6.0 built. SPAU/SPDD is done and is relaible
2) Table logging in turned on in the source release since that date.
3) The tools keeps flushing the delta between the two systems. Here the tool manages the strucure changes and others for standard SAP objects. For custom tables or any special add on's, you have to provide the list.
4) A 2 - 6 hour outage for the last delta to be flushed out.

But this is still in testing phase in SAP Labs.


Best Technical Practices for SAP Landscape Maintenance
In this post I would like to share my experience on Proactive measures to be taken in a SAP Landscape.
The recommendations are slightly biased towards technical aspects of SAP; hence I would request the readers to only apply them in your own environment after a careful analysis. Thanks for it!
You are most welcome for your suggestions and corrections.

I would like to divide the main topic into 7 sub-components

* Proactive Measures
* Performance Optimization
* Disaster Recovery
* Problem-Solution Approach
* Onsite vs Off-shore Support Delivery
* Upgrades
* System copies


Proactive Measures
This section emphasizes on key recommendations on preparation aspects to be satisfied for ever-challenging and ongoing performance tuning by Basis, ABAP and Functional Leads. These recommendations might give an impression that they would involve efforts during the initial stages of practice. But indeed this practice yields fruitful results once put in practice. Patience definitely pays! I would also be helping out in getting the intended results indeed by giving examples and code wherever necessary. Before I start just want to ensure the readers that this model works PERFECTLY if basis team works at a client place or close to the client environment. In off-shore support model, implementing these recommendations would be slightly tough. Though, the decision is left for you to decide which of them best suits your needs. I believe now I am ready go into topic.

1. Technical review Tracker:
This can be a Power-point presentation or a Word document depicting the technical aspects based on which the tuning, enhancements and changes can be decided. This presentation is updated twice in month by the Basis Team and will be reviewed by the Technical lead, Functional leads and other influencing persons. The objective of periodic reviews of this presentation would be to achieve the intermediate goals set during the review meetings. Let me be more specific about this presentation. The following are the slides that can be used for this presentation. Keep in mind that this presentation is unique to each SAP system landscape like ECC, BI, PI, SOLMAN, CRM, SRM etc.
1. Hardware Details
2. CPU Utilization of Primary, Additional Apps, Database servers
3. System landscape and the transport route
4. Database space management
5. Buffer Invalidations
6. Top Background Jobs
7. EarlyWatch Recommendations

I would give a brief about each of the above on how it works. H/W details slide shows hostname, RAM, physical/virtual CPUs on the LPAR/Server, CPU entitled capacity, IP address etc. CPU Utilization slide would help in identifying the CPU power bottlenecks and under utilization of CPU. This helps in identifying the SAP components causing the maximum utilization there by optimizing the total performance of the server and also reorganizing the logon groups. This helps especially during month-end releases and COPA activities which imposes maximum impact on the total server performance. Now, database space management slide. As we all know this is very crucial for the overall performance of the SAP server.



Script for killing all processes matching a search string!

ps -e -o ":%p:%a" | grep | cut -d":" -f2|while read p
do
echo processes $p
kill -9 $p
done

0 comments
NetWeaver Installation Part II
Posted by Venky on at 8:21 AM


32. What are sapadm, adm and webadm used for?
A. sapadm is used for Central monitoring services. adm is dedicated to Diagnostic agent installation. webadm is created to use web administration interface for Internet Communication Manager (ICM) and web dispatcher.

33. How to prepare SAP system for NWDI integration?
A . If you select check box Prepare SAP system for NWDI Integration on the screen NWDI Landscape, SAPinst copies all SCAs belonging to the software units that you installed to the global transport directory.

The SAP NetWeaver Development Infrastructure has a set of services that provide central storage and distributed versioning of Java source code, plus centralized build services and a predefined change management process that control the folow of the software through the landspace. To integrate your SAP system into the development landscape, NWDI requires all installed software components to be available in the file system.

34. How do you plan the Switchover Cluster?
A. To reduce unplanned downtime for your SAP system by setting up a switchover cluster. This setup installs critical software units - know as "single points of failure" (SPOFs) - across multiple host machines in the cluster. In the event of a failure on the primary node, proprietary switchover software automatically switches the failed software unit to another hardware node in the cluster.
Manual intervention is not required. Applications accessing the failed software unit might experience a short delay but can then resume processing as normal.
Switchoever clusters also have the advantage that you can deliberately initiate switchover to free up a particular node for planned system maintenance. Switchover solutoins can protect against hardware failure and operating system failure but not against human error.

35. What a switchover cluster consists of?
A.
1. HW cluster of two or more physically separate host machines to run multiple copies of the critical software units, in an SAP system the SPOFs referred to above.
2. Switchover software to detect failure in a node and switch the affected software unit to the standby node, where it can continue operating.
3. A mechanism to enable application software to seamlessly continue working with the switched software unit - normally this is achieved by virtual addressing.


36. How do you manage the creation of operating systems users during the SAP installation in the case of NIS (Network Information System)?
A.
SAPinst checks if the required services are available on the host and creates them if necessary. See the log messages about the service entries and adapt the network-wide (NIS) entries accordingly.
SAPinst checks the NIS users, groups, and services using NIS commands. However, SAPinst does not change NIS configurations.
For a distributed or a high-availability system, its recommended that you distribute account information (operating system users and groups) over the network by using NIS.
Note: If you use NFS-V4 file system, you have to create the ora user on the NFS server.
37. Directory structure of ABAP+Java System.

AIX tools for SAP Administrators Set 4


Some really funky hardware-looking problems can be fixed by draining
the NVRAM battery for 5 minutes, and then reinstalling the microcode.
We used to do this on some IBM RT's in Cincinnati, and a recent poster
to AIX-L indicates that it's still useful in some situations.

Why don't you try using iptrace and ipreport to see the behavior of your
telnet sessions ??

/usr/bin/uname -M
Gets the machine type

acledit

AIX tools for SAP Administrators Set 3


savevg -v -n -9 / _rootvg.img rootvg
This can be used to build a NIM installable image to recover your systems

alternatively, the command line call for a mksysb to tape (to include map
and exclude files) is /usr/bin/mksysb '-m' '-e' '-i' /dev/rmt0

Finding which lpp contains a file:
lslpp -w /usr/sbin/umount

AIX Standalone Diagnostics
http://www.ibm.com/developerworks/wikis/display/WikiPtype/IBM+Standalone+Diagnostics

Show real memory
lsattr -El sys0 | grep realmem
lsattr -El mem0

Example
user@(/home/user)$ lsattr -El sys0
SW_dist_intr false Enable SW distribution of interrupts True
autorestart true Automatically REBOOT system after a crash True
boottype disk N/A False
capacity_inc 0.01 Processor capacity increment False
capped false Partition is capped False
conslogin enable System Console Login False
cpuguard enable CPU Guard True
dedicated false Partition is dedicated False
ent_capacity 4.00 Entitled processor capacity False
frequency 1056000000 System Bus Frequency False
fullcore false Enable full CORE dump True
fwversion IBM,SF240_358 Firmware version and revision levels False
id_to_partition 0X800001D372E00006 Partition ID False
id_to_system 0X800001D372E00000 System ID False
iostat true Continuously maintain DISK I/O history True
keylock normal State of system keylock at boot time False
log_pg_dealloc true Log predictive memory page deallocation events True
max_capacity 6.00 Maximum potential processor capacity False
max_logname 9 Maximum login name length at boot time True
maxbuf 20 Maximum number of pages in block I/O BUFFER CACHE True
maxmbuf 0 Maximum Kbytes of real memory allowed for MBUFS True
maxpout 8193 HIGH water mark for pending write I/Os per file True
maxuproc 1024 Maximum number of PROCESSES allowed per user True
min_capacity 0.10 Minimum potential processor capacity False
minpout 4096 LOW water mark for pending write I/Os per file True
modelname IBM,9117-570 Machine name False
ncargs 8 ARG/ENV list size in 4K byte blocks True
nfs4_acl_compat secure NFS4 ACL Compatibility Mode True
pre430core false Use pre-430 style CORE dump True
pre520tune disable Pre-520 tuning compatibility mode True
realmem 16777216 Amount of usable physical memory in Kbytes False
rtasversion 1 Open Firmware RTAS version False
sed_config select Stack Execution Disable (SED) Mode True
systemid IBM,0265151DA Hardware system identifier False
variable_weight 64 Variable processor capacity weight False


See if you AIX system's hardware is CHRP (some sort of PowerPC reference
platform spec, I believe) :
bootinfo -p
chrp


AIX tools for SAP Administrators Set 2


List subsystem, this shows the subsystems that are active and inactive along with their PIDs
lssrc -a

This starts sendmail
startsrc -s sendmail -a "-bd -q30m"


lsps is used to list the characteristics of paging space

Turning off ip forwarding:
/usr/sbin/no -o ipforwarding=0

Detailed info about a specific error (errors are stored in NVRAM on AIX)
errpt -a -jE85C5C4C


Checking the fileset installation
lslpp -l

show LPAR configuration
lparstat -i or prtconf

Run the ssadiag against the drive and the adapter and it will tell you if it fails or not. Then if its a hot plugable it can be replaced online.

Backup to tape:
env - /usr/bin/mksysb '-m' '-i' '-X' /dev/rmt0
The "env -" is because some sort of environment variable can confuse
mksysb, making it error out instead of doing your backup
There's also "smitty mksysb"

You can create an image using the savevg command i.e.
savevg -v -n -9 / _rootvg.img rootvg


AIX tools for SAP Administrators Set 1


VM Utilization
vmstat 3 5

Checking for CLOSE_WAITs
CLOSE_WAIT Indicates passive close. Server just received first FIN from a client.
netstat -a | grep CLOSE_WAIT | wc -l

ps auxxxx
The -a option tells ps to list the processes of all users.
The -u option tells ps to provide detailed information about each process.
The -x option adds to the list processes that have no controlling terminal, such as daemons, which are programs that are launched during booting (i.e., computer startup) and run unobtrusively in the background until they are activated by a particular event or condition.

Top 10 processes
svmon -Pvt 10 | more

Top 10 users
svmon -Uvt 10 | more

ipcs -pmb | grep

ipcs

topas -P
Topas Monitor for host: vws031 Interval: 2 Tue Feb 9 12:15:44 2010

DATA TEXT PAGE PGFAULTS
USER PID PPID PRI NI RES RES SPACE TIME CPU% I/O OTH COMMAND
decadm 819304 1196242 63 20 40710 11971 43398 41:22 0.2 0 1 disp+wor
root 495666 1 60 20 709 31 2466 28:57 0.1 0 0 kulagent
in0437891966236 1855502 60 20 740 109 740 0:00 0.1 0 3 topas
cipadm 700444 737466 82 20 88459 471 138722 61:15 0.0 1 1 jlaunch
cipadm 352362 737466 82 20 8769 471 18335 43:44 0.0 0 0 jlaunch
cipadm 753914 737466 82 20 12576 471 29115 31:51 0.0 0 0 jlaunch
decadm 745700 1196242 60 20 932 1160 1343 5:57 0.0 0 0 gwrd
root 1306808 1 60 20 238 360 422 1:42 0.0 0 25 saphoste
sapadm 1818770 1 60 20 52494 2446 232104 22:36 0.0 0 0 sapstart

To show the version of the software
lslpp -al | grep

To show the OS version
oslevel -r

VM tuning
vmo

Stopping lan and enabling it

ifconfig en2 down detach
chdev -l ent2 -a jumbo_frames=yes
chdev -l en2 -a mtu=9000
chdev -l en2 -a state=up

NFS tuning parameters, good to check on if you're getting badcalls in nfsstat
nfso -a

nfsstat -m shows great info about full set of NFS mount options

To check for software problems:
lppchk -v
lppchk -c
lppchk -l


Identifying server node ID in multi-server node environment


If performance issues occur on a portal it might help to determine which server/dispatcher is used to answer requests.
Use the doc below to help you identify the dispatcher/server

Logon to a portal URL https:///irj/portal OR another portal being used
As soon as you have logged on type in the browser window javascript:alert(document.cookie)
This will produce a box similar to the screen shot below






JSESSIONID is a unique ID that you can check in the /usr/sap/SID/JC00/j2ee/cluster/instance.properties this will indicate the server0 used and the dispatcher, particularly useful if multiple web dispatchers are used for Load balancing.