enqt opcode 39

75    /*--- operation code --------------------------------------   

76    #define OPCODE_NOP         cU(' ')   

77    #define OPCODE_RESET_STATISTICS  cU('-')   

78    #define OPCODE_STATISTICS  cU('0')   

79    #define OPCODE_ENQUEUE     cU('1')   

80    #define OPCODE_DEQUEUE     cU('2')   

81    #define OPCODE_DEQUEUE_ALL cU('3')   

82    #define OPCODE_CLEANUP_USER cU('4')  

83    #define OPCODE_REMOVE_GEN  cU('5')  /* generic remove */      

84    #define OPCODE_GETUSER     cU('7')   

85    #define OPCODE_CLRSTAT     cU('8')   

86    #define OPCODE_GETSTAT     cU('9')   

87       

88    #define OPCODE_ACTUAL_FILE   cU('A')   

89    #define OPCODE_BCK_ON        cU('B')   

90    #define OPCODE_COMPRESS      cU('C')   

91    #define OPCODE_DUMP          cU('D')   

92    #define OPCODE_BCK_OFF       cU('E')   

93    #define OPCODE_ENQUEUE_OFF   cU('F')   

94    #define OPCODE_GET_ENQID     cU('G')   

95    #define OPCODE_GET_HRES_TIME cU('H')   

96    #define OPCODE_SET_ENQID     cU('J')   

97    #define OPCODE_GET_STAMP     cU('N')   

98    #define OPCODE_STATE_TRANSFER cU('O')   

99    #define OPCODE_ENQUEUE_PERM  cU('P')  

100    #define OPCODE_QUERY         cU('Q')  

101    #define OPCODE_REMOVE        cU('R')  

102    #define OPCODE_ENQUEUE_STATE cU('S')  

103    #define OPCODE_ENQUEUE_ON    cU('T')  

104    #define OPCODE_VERIFY        cU('V')  

105    #define OPCODE_LOG_ON        cU('X')  

106    #define OPCODE_DUMP_JSON     cU('d')  

107    #define OPCODE_LOG_ON_UNAME  cU('x')   /* lower case */  

108    #define OPCODE_LOG_OFF       cU('Y')  

109    #define OPCODE_LOG_OFF_TRUNC cU('y')   /* lower case */  

110    #define OPCODE_LOG_ON_REPL   cU('Z')  

111    #define OPCODE_LOG_OFF_REPL  cU('z')   /* lower case */

Semaphore statistics

To active it

transaction ST02 ->

buttons 'Detail Analysis Menue' ->

'Semaphores'.

The measurement is switched on automatically.

To switch it off

menu 'Settings' ->

'Monitoring Off'.

To download the list

OK-code '%PC'

shared memory owner

How to check the shared memory owner?

1. showipc

2. get the key number and find the address, for example:

> showipc 01 | grep 51

OsKey: 20151 0x00004eb7 Semaphore Key: 51

the address is 4eb7

3. call the ipcs with the address, for example:

> ipcs | grep 4eb7

0x00004eb7 2818130 loladm 740 1

semd

Op: Rq=1,Rl=2,Sh=3,In=4,InM=6,RmM=7,Clo=8,RmLM=9,Exit=10,AllowReInit=11 :10

3: Show the status of the semaphore.

4: Command "initialize".

10: Exit the semd program.

ex:

#!/bin/bash

SEMA=3; echo "4 $SEMA -1 3 10" > options; semd pf=INSTANCE_PROFILE < options

saposcol clean shared memory

saposcol -d

Collector > clean

Collector > quit

saposcol -k to stop the collector

saposcol -d (before restarting)

Collector > leave (You should get a message Shared memory deleted)

Collector > quit

cd /usr/sap/tmp

mv coll.put coll.old (rename the file - it stores the saposcol history data)

saposcol (run it again)

ICM - Web disaptcher

Csi* = Content Filter

HttpSrvHdlRequest = URL requested

session context-----------------------

DpFindWebContext

IcmSendRq called with session key:

----------------------------------------

JNCMINodeGet: node found

HttpJ2EEDispatchLb

HttpJ2EETriggerServer: request dispatched to server XXXXXX

3223

rdisp/wp_ca_blk_no

rdisp/appc_ca_blk_no

rdisp/tm_max_no

rdisp/max_comm_entries

what to do at the moment of incident

Follow the below steps and try to collect the greatest number of
possible evidences:

1) Analyse from the Operating System side:
a. Is it possible to connect to the OS level?
b. Is there any CPU, IO, memory bottleneck (check with OS tools)?


2) Collect the queue status of SAP dialog instance:
a. dpmon pf= l
b. sapcontrol -nr -function ABAPGetWPTable

You should get the queue status of the instace during around 10 or
15 minutes to check if the work process are changing the status in that
timeframe or they are really hanged. For the sapcontrol you can use the
option "-repeat " (call times (-1=forever) with sec
delay) and for dpmon see the scrip of note 675778.


3) Collect queue statistics:
a. dpmon pf= d
b. sapcontrol -nr -function GetQueueStatistic

4) Collect enqueue statistics (ASCS)
a. sapcontrol -nr -function EnqGetStatistic
(as from 710)
b. transaction SM12 -> Extras -> Statistics


5) Collect process status of the instance:
a. sapcontrol -nr -function GetProcessList

If you are not sure where the problem is you should collect this
information for all available instances.


6) Collect the gateway statistics:
a. gwmon


7) Increase the trace level of work process:
a. kill -USR2
b. sapntkill -USR2 (for windows enviroment)

You can get the PID of the work process in SM50, dpmon or sapcontrol
(see item 2). This will increase the dispatcher/work process trace file
and provide further data in the trace. Call this command twice for the
PID.


8) Always SAVE the work folder with the traces:
You do find the instance work folder in
/usr/sap///work
Every incident you must save the work folder of this instance before
the restart or soon as after the start up.


9) Get the timestamps (date and hour) of all events;


10) Collecting the SAP instance parameters:
a. sapcontrol -nr -function ParameterValue
b. sappfpar all pf=LOL_DVEBMGS01_lolserver
c. transaction rspfpar

Work process status (DP_WAIT_REASON)

Every work process can get status as per what it is doing.

DP_WAIT_REASON is an enumeration of all possible wait reasons.

NO_WAITING = 0, Session does not wait
WAITING_FOR_DEBUG_IO = 1, Session waits for debugger input
WAITING_FOR_RFC = 2, Session waits for RFC response
WAITING_FOR_ENQUEUE = 3, Session waits for enqueue response
WAITING_FOR_VB = 4, Session waits for update response
WAITING_FOR_SPO = 5, Session waits for spool
WAITING_FOR_ADM = 6, Session waits for adm message response
WAITING_FOR_NO = 7, Session waits for number range buffer response
WAITING_FOR_GUI = 8, Session waits for gui input
WAITING_FOR_PRIV = 9, Session is hold because of private memory
WAITING_FOR_VB_ADM = 10, Session waits for update admin message response
WAITING_FOR_SERV_LIST = 11, Session reads the server list
WAITING_FOR_GWMON = 12, Session waits for gateway monitor response
WAITING_FOR_SNC = 13, Session waits for return of blocking snc call
WAITING_FOR_USER_LOCK = 14, Session is locked for a dedicated user
WAITING_FOR_TIMEOUT = 15, Session sleeps
WAITING_FOR_MS = 16, Session waits for a message to be send
WAITING_FOR_OS = 17, Session waits for execution of an operation system command
WAITING_FOR_MSGOPCODE = 18, Session waits for execution of an msg server action
WAITING_FOR_ARFC = 19, Session waits for ARFC execution
WAITING_FOR_RPC = 20, Session waits for response of an remote action
WAITING_FOR_VB2 = 22 Session waits for update 2 response

ST03G not show the instance

The configuration of the transaction ST03G is 100% user specific and
this is not related to the user authorization, but every configuration
for every specific user is stored in the table SGLWLSLIST.

So that every specific user does need to setup there own configuration
in the transaction ST03G. You can use the report SAPWL_CPSGLWLSLIST to
copy an existing user configuration.

sapcontrol webmethods list

Webmethods list:

==> Start-----------------------------------------------------------
BASIC: sapcontrol -nr -function Start
To start up the SAP instance


==> Stop [softtimeout sec]-----------------------------------------
BASIC: sapcontrol -nr -function Stop
This is a faster method stop the SAP instance.
This will kill the SAP Instance using OS commands.
The processes of the instance are terminated with a signal 9 (kill -9).
The instance shared memory is also removed (cleanipc remove).
The file kill.sap in the instance work folder is deleted.


==> Shutdown------------------------------------------------------
BASIC: sapcontrol -nr -function Shutdown
A normal shutdown of the instance
The SAP instance will be stopped in normal process.


==> RestartInstance [softtimeout sec]-----------------------------


==> StopService---------------------------------------------------


==> StartService --------------------------------------------


==> RestartService------------------------------------------------


==> ParameterValue []----------------------------------
BASIC: sapcontrol -nr -function ParameterValue
Show the instance parameters and the actual parameter value.
Without the option you do list all parameters of the instance.

The same output of: RZ11


==> GetStartProfile----------------------------------------------
BASIC: sapcontrol -nr -function GetStartProfile
Show all lines of the instance start profile

The same output of: RZ10


==> GetTraceFile------------------------------------------------


==> GetAlertTree------------------------------------------------


==> GetAlerts---------------------------------------------------


==> GetEnvironment----------------------------------------------
BASIC: sapcontrol -nr -function GetEnviroment
This will return the enviroment variable to the
actual user.

The same output of: similar comand "env" in the UX system


==> GetVersionInfo---------------------------------------------
BASIC: sapcontrol -nr -function GetVersionInfo
Return the actual version information about each service
from that instance.

The output have the format below:
Filename = File path fo that service
Versioninfo = version and patch level
Time = Date of the compilation


==> GetQueueStatistic-----------------------------------------
BASIC: sapcontrol -nr -function GetQueueStatistic
Show the Queue information of the instance.

The same output of: SM51 -> Goto -> Queue Information

The output have the format below:
Typ = Type of the queue (ex: DIA, UPD, BTC, etc)
Now = Actual number of waiting requests
High = The bigger number of requests in the queue
Max = Limit of the queue
Writes =
Reads =


==> GetProcessList-------------------------------------------
BASIC: sapcontrol -nr -function GetProcessList
This will return all services from the instance and the actual status of
each one.

The output have the format below:
name = Program name
description = Service Description
dispstatus = GRAY = unavailable, GREEN = running, YELLOW = statring
textstatus = Information about the service
starttime = Started at
elapsedtime = Time running
pid = Process ID in the Operating System


==> GetInstanceProperties------------------------------------


==> ListDeveloperTraces--------------------------------------
BASIC: sapcontrol -nr -function ListDeveloperTraces
List the developer traces found in the work folder

The same output of: ST11

The output have the format below:
filename =
size =
modtime =


==> ReadDeveloperTrace -----------------


==> ListLogFiles---------------------------------------------
BASIC: sapcontrol -nr -function ListLogFiles
List all logs from the instance.

The same output of: ST11

The output have the format below:
filename = the file of the trace
size = size of the file
modtime = last modification of the trace
format = usually text


==> ReadLogFile [ [ [ []]]]


==> AnalyseLogFiles [] [] [ ]


==> OSExecute ----


==> SendSignal -------------------------------


==> GetSystemInstanceList-----------------------------------
BASIC: sapcontrol -nr -function GetSystemInstanceList

The output have the format below:
hostname =
instanceNr =
httpPort =
httpsPort =
startPriority =
features =
dispstatus =
linux, 0, 50013, 0, 2, ABAP|GATEWAY|MESSAGESERVER|ENQUE|ICMAN, YELLOW


==> StartSystem [ALL|SCS|DIALOG|ABAP|J2EE|LEVEL []]


==> StopSystem [ALL|SCS|DIALOG|ABAP|J2EE|LEVEL [[] [softtimeout sec]]]


==> RestartSystem [ALL|SCS|DIALOG|ABAP|J2EE|LEVEL [[] [softtimeout sec]]]


==> AccessCheck ---------------------------------


==> ABAPReadSyslog-----------------------------------------
BASIC: sapcontrol -nr -function ABAPReadSyslog
List the SAP syslog entried.

The same output of: SM21

The output have the format below:
Time =
Typ =
Client =
User =
Tcode =
MNo =
Text =
Severity =
2011 02 12 20:26:46, D0, 000, SAPSYS, , A10, SAP Basis System: Initialization complete, GREEN


==> ABAPReadRawSyslog--------------------------------------


==> ABAPGetWPTable-----------------------------------------
BASIC: sapcontrol -nr -function ABAPGetWPTable
List the work process table list.
The same output of: SM50

The output have the format below:
No, Typ, Pid, Status, Reason, Start, Err, Sem, Cpu, Time, Program, Client, User, Action, Table
0, DIA, 9254, Wait, , yes, , , 0:04:50, , , , , ,
1, DIA, 9255, Wait, , yes, , , 0:00:28, , , , , ,
2, DIA, 9258, Wait, , yes, , , 0:00:21, , , , , ,


==> J2EEControlProcess
==> J2EEGetProcessList
==> J2EEGetThreadList
==> J2EEGetSessionList
==> J2EEGetCacheStatistic
==> J2EEGetApplicationAliasList
==> J2EEGetVMGCHistory
==> J2EEGetVMHeapInfo
==> StartWait
==> StopWait
==> WaitforStarted
==> WaitforStopped

memory DUMPS

The memory for the user context is used up?:
STORAGE_PARAMETERS_WRONG_SET
SYSTEM_ROLL_IN_ERROR
TSV_TNEW_BLOCKS_NO_ROLL_MEMORY
TSV_TNEW_PAGE_ALLOC_FAILED
TSV_TNEW_NO_ROLL_MEMORY

The allocated areas in the shared memory (EM and PXA) violate the OS specific restrictions
PXA_NO_SHARED_MEMORY

Memory boottleneck durring an operation in the database interface
DBIF_RTAB_NO_MEMORY
DBIF_RSQL_NO_MEMORY

SAP paging memory is used up
SYSTEM_NO_MORE_PAGING
TSB_TNEW_PAGE_ALLOC_FAILED

If ztta/max_memreq_MB is set too low
SYSTEM_NO_ROLL

Performance indicator for ABAP (ST03)

Below you do find the thresholds for each item from ST03

- Wait time < 10% of the rsponse time
- Average roll-in time < 20ms
- Average roll-wait time < 200ms
- Average load (and generation) time < 10% of response time (<50ms)
- Average database request time < 40% of (respose time - wait time)
- Average CPU time < 40% of (resposne time - wait time)
- Average CPU time not much less than processing time
- Average response time depends of requirements. There is no general rule

sapcpe

The sapcpe is a tool responsible to copy the kernel folder from

/usr/sap//SYS/exe/run

to

/usr/sap///exe

Important profile paramters for the sapcpe

DIR_CT_RUN = $(DIR_EXE_ROOT)/run

DIR_EXECUTABLE = $(DIR_INSTANCE)/exe

Shared Memory Areas

1 "System administration "

2 "Disp. administration tables "

3 "Disp. communication areas "

4 "statistic area "

5 "SCSA area "

6 "ABAP program buffer "

7 "Update task administration "

8 "Paging buffer "

9 "Roll buffer "

10 "Pool "

11 "Factory calender buffer "

12 "TemSe Char-Code convert Buf. "

13 "Alert Area "

14 "Presentation buffer "

15 "Document administration "

16 "Semaphore activity monitoring"

17 "Roll administration "

18 "Paging adminitration "

19 "Table-buffer "

20 "Unused1 "

21 "Unused2 "

22 "Unused3 "

23 "Unused4 "

24 "Unused5 "

25 "Unused6 "

26 "Unused7 "

27 "Unused8 "

28 "Unused9 "

29 "Unused10 "

30 "Taskhandler runtime admin. "

31 "Dispatcher request queue "

32 "unknown segment "

33 "Table buffer, part.buffering "

34 "Enqueue table "

35 "Sap-Communication BK/EDI "

36 "Performance monitoring "

37 "Fast SEM ADM on MPE & AS400 "

38 "Event ADM on AS400 "

39 "Queue ADM on AS400 "

40 "Pool for database buffers "

41 "DB statistics buffer "

42 "DB TTAB buffer "

43 "DB FTAB buffer "

44 "DB IREC buffer "

45 "DB short nametab buffer "

46 "DB sync table "

47 "DB CUA buffer "

48 "Number range buffer "

49 "Spool admin (SpoolWP+DiaWP) "

50 "Pool 50 "

51 "Extended memory admin. "

52 "Message Server buffer "

53 "Extended memory test tool "

54 "Export/Import buffer "

55 "Spool local printer+joblist "

56 "Application statistics "

57 "Profilparameter in shared mem"

58 "Enqueue ID for reset "

59 "Enqueue table extension AS400"

60 "Enqueue table addition AS400"

61 "Extended memory only on AS400"

62 "Memory pipes "

63 "ICMAN shared memory "

64 "Online Text Repository Buf. "

65 "Export/Import Shared Memory "

66 "Enqueue Replication Shadow "

67 "ICRouter HTTP Client Table "

68 "JControl Administration "

69 "JControl Administration "

70 "java area "

71 "ITS Events "

72 "ITS Mutexe "

73 "CCMS Extended Alert Area "

74 "VMC Lite Admin "

75 "VMC Lite ES "

76 "Ext. Segment Administration "

77 "Ext. Memory Administration "

78 "Ext. Global Memory Admin "

79 "JStart static instance info "

80 "JStart dynamic data "

81 "Security Audit Log "

82 "Web Disp EUID Table "

83 "ABAP Coverage Analyzer "

101 "Performance monitoring "

102 "Performance monitoring V01.0 "

103 "Global spool storage "

108 "Network monitoring "

121 "SCSA area "

sappfpar

Some hits from sappfpar:

1. to list all instance parameters

sappfpar all pf="INSTANCE PROFILE"

2. to check the memory parameter in the instance and check the recognized parameters

sappfpar check pf="INSTANCE PROFILE"