SecureDBA

Kevin Sheehan's Podcast on Secure Application Infrastructure & Federal Data Center Migration

"The Federal Data Center Consolidation Initiative calls upon agencies to develop and implement plans for consolidating, streamlining and modernizing their datacenter operations.  Based on his experience migrating mission-critical applications to new, state-of-the-art facilities, Agilex’ Kevin Sheehan shares important lessons learned for meeting this mandate by creating a more agile and secure infrastructure.  A key focus is in the use of standard configurations and common platforms to streamline the certification & accreditation process while improving application security and reliability.  He also weighs the pros and cons of emerging technologies, such as virtualization and Oracle Exadata." ¹

You can find the Agilex Podcast here.

¹ Agilex Technologies Podcast (July 26, 2010), viewed July 29, 2010, http://agilex.podbean.com/2010/07/26/migrating-to-more-secure-and-agile-application-infrastructure-with-agilex’-kevin-sheehan/ 

The Perils of 'Assistants' and Other Black Boxes

A few weeks back I published a post called Security patching can make you LESS Secure! Redux which reports that upgrading your 10.2.0.3 Oracle database to 10.2.0.4 causes the WKSYS password to be reset to the default value and _trace_files_public to become unset, meaning set to TRUE.

A little more research revealed that the problem lies in the Database Upgrade Assistant or DBUA. If you perform a manual upgrade, neither of the above two issues occurs. The bottom line is that you should never run a blackbox 'assistant' when you can simply run a few scripts. Who knows what that blackbox is doing. In this case, the only thing the DBUA 'assisted' with was making your database less secure.

The manual upgrade is easy and fully documented in the patch readme.  After installing the software simply:

SQL> STARTUP UPGRADE
SQL> SPOOL patch.log
SQL> @?/rdbms/admin/catupgrd.sql
SQL> SPOOL OFF

Review the patch.log for issues and then:

SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP
SQL> @?/rdbms/admin/utlrp.sql

You should not need DBUA for these simple tasks.

By the same logic you should consider not running DBCA, the Database Creation Assistant or allow the installer to create a database for you. This is easily scripted for most databases.

Security patching can make you LESS secure! Redux

I started off this site in October 2007 with a post talking about how upgrades and patches can really negatively impact the security posture of your infrastructure. Well here it is nearly two years later and the problem persists.

We'll pick on Oracle here but the problem is common among most software vendors.  Let's take the recent Oracle Critical Patch Update for April 2009. If you are running a 10.2.0.3 database, you'll have to upgrade to 10.2.0.4 (or higher) to install the patch since 10.2.0.3 is no longer supported.

So far we have identified two issues with the 10.2.0.3 to 10.2.0.4 upgrade. The first item applies only if you have Ultrasearch installed. Installation of Ultrasearch will create a schema called WKSYS. In a properly secured database, this schema should have its default password changed to a strong password and the account should be locked, expired and audited. Assuming your database is so secured, you will find after running the 10.2.0.4 Database Upgrade Assistant, that the WKSYS account is unlocked and the password has been reset to match the name of the account, ouch!

The other issue we have found so far in every 10.2.0.3 to 10.2.0.4 upgrade is that _trace_files_public gets reset to TRUE.

It should be noted that in both cases the upgrade caused the issue. The actual CPU patch had no impact on the database security posture.

We are not yet sure if these issues are generic or platform specfic. We found the issues on Solaris Sparc 64-bit.

The moral of the story is you need to scan your infrastructure after any changes. Again, we recommend AppSentry from Integrigy because we work a lot with the Oracle E-Business Suite where this tool is unmatched. But it works great for plain old Oracle databases as well.  See Integrigy's Web Site  for details and Stephen Kost's Oracle Security Blog at the same site is always a worthy read.

Is your Oracle 10.2.0.2 Databse on Solaris Secure?

If you are running Oracle 10.2.0.2 on Solaris SPARC 64-bit,  the January Critical Patch Updates for Database and Middleware Products reports N/A for patch availability. I think this is kind of curious that other versions of Solaris are patched but not this one yet there are patches for this version of Solaris on the previous and next releases of the database. My recommendation would be to upgrade to 10.2.0.3 and then apply the January CPU to make sure your are covered.

Hardening the Oracle 11g Database - Initial Steps

The following is a basic set of hardening guidelines for an Oracle 11g database along with some scripts you may find useful. This list is by no means complete. It does not cover file permissions, authentication controls and user profiles, encryption, grants or auditing but it is a good place to start.

Or perhaps takes Slavic's advice from his comment on this post and start with  Oracle's Checklist for Database Security.  It gives a broader, though perhaps less detailed view, and covers some of the topics I have left out (for now). Also check out Slavik's Musings on Database Security Blog.

1. Put the database server behind a firewall. See prior post  Wall It Off!  for a discussion of the importance of a firewall, even for trusted users on your Intranet.
2. Apply latest security patches for OS and harden the operating system.
3. Install only the components needed. When installing the Oracle software, Spatial, OLAP, Data Mining, and Real Application Testing are all installed by default. Change the default behavior if you do not need these components. The MDSYS schema used by Spatial has been the target of security vulnerabilities in the past.
4. 11g defaults to installing Advanced Security; this is a default that should be accepted assuming you have the license.
5. Change the default name of the Listener (LISTENER) as well as the default port (1521).
6. Do not install the sample schemas which are installed by default.
7. When running the Database Configuration Assistant, select the Custom option so that you have the ability to de-select any unneeded components.
8. Do take the default security settings, new for 11g, that enable auditing and a more hardened default profile. This setting also revokes CREATE EXTERNAL JOB from PUBLIC, sets 07_DICTIONARY_ACCESSIBILITY and REMOTE_OS_ROLES to FALSE as well as setting values for PASSWORD_GRACE_TIME, PASSWORD_LOCK_TIME and PASSWORD_LIFE_TIME to 7, 1, and 180 respectively rather than their previous default of UNLIMITED. This is a great move by Oracle.
9. More good news. With 11g, a new initialization parameter called SEC_RETURN_SERVER_RELEASE_BANNER is available that controls the display of Oracle version information when clients connect to the database. It even defaults to FALSE (do not display), another good move by Oracle.
10. Even more good news. With 11g, a new initialization parameter called SEC_CASE_SENSITIVE_LOGON is available to allow case sensitive passwords. It even defaults to the TRUE setting, meaning case sensitivity is supported. This can greatly increase password complexity and make it harder to use brute force attacks to guess passwords. Make use of this new feature when setting passwords.
11. Exit the installer. Prior versions of Oracle left encrypted passwords in the install logs which could be easily decrypted by hackers. 11g appears to do a better job of this, instead displaying XXXXXXX or *********** . However, I still consider it a best practice to change passwords at the command line rather than through a tool. Make sure to use the "password <username>" syntax rather than the "alter user <username> identified by <password>" syntax since the former encrypts the password over the network and the later does not (at least in prior versions).
12. Change the default passwords for all the locked accounts. These passwords can be made very , very strong since you will never use them.
13. Also change the passwords of SYS and SYSTEM. Again this is just a precaution in case the installer left a remnant of the password you entered during the installation.
14. If you configured Enterprise Manager (OEM) then also change the SYSMAN and DBSNMP account passwords using the appropriate procedure for those accounts in order not to break your OEM installation.
15. Next you'll want to scan for default or weak passwords. I used to maintain my own script for this but now I rely on Alexander Kornbrust's, founder of Red Database Security GmbH . This is the best tool I've found and Alexander makes it freely available at: Oracle Password Checker (Cracker) .
16. With 11g, Oracle also supplies a DBA_USERS_WITH_DEFPWD view. Selecting * from this view will return USERNAMEs with default passwords. This is nice to have but I still recommend Alexander's script instead since it also checks for weak passwords.
17. You may be asking why all this checking for default and weak passwords when 11g does a good job of this? First, you may be upgrading from a previous, less secure version and have some old defaults. Second, DBAs are not infallible and sometimes create weak passwords. Scan regularly!
18. Next we'll check for demo accounts. If you've followed the above instructions you should have none but it you are upgrading an existing database or forgot to change the default behavior of the installer, now is the time to check. I use Randy Giefer's Demo Account Check SQL Script which you can download here. Simply run the script and drop any demo accounts it finds.
19. Next we need to harden the database listener. Oracle created an EXTPROC entry in your listener.ora configuration file whether you needed one or not. If you do not need to make external procedure calls from the database then the EXTPROC listener is not needed and should be removed. External procedure calls can give an attacker access to your operating system from within the database. Removing this capability eliminates this attack vector.The entry will look something like this: (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC<your-non-default-listener-port>)) . Simply backup your current listener.ora, remove the above line, and bounce the listener.
20. Next you'll want to set administration restrictions on your listener. This will prevent the use of the SET command and require any listener configuration changes to be done by a user with write permissions on the listener.ora file. Simply save a backup of the listener.ora file, add the following line: ADMIN_RESTRICTIONS_LISTENER_<your-non-default-listener-name>=ON and then bounce the listener.
21. Listener logging is turned on be default; however, it is generally a best practice to explicitly set values, even if it is a default. To explicitly set this parameter, save a backup of the listener.ora file, add the following line: LOGGING_LISTENER_<your-non-default-listener-name>=ON and then bounce the listener.
22. The CIS benchmark recommends setting INBOUND_CONNECTION_TIMEOUT. This parameter specifies the time, in seconds, that a client must complete its connection attempt after the initial network connection has been acknowledged. The recommendation is to set it to the lowest value possible where none of your client connections are timing out. By default this parameter is not set. To set this parameter, save a backup of the listener.ora file, add the following line: INBOUND_CONNECTION_TIMEOUT_<your-non-default-listener-name> n where n is the number of seconds in which the client must complete its connection, and then bounce the listener.
23. The CIS benchmark recommends setting SQLNET.INBOUND_CONNECTION_TIMEOUT. This parameter is similar to INBOUND_CONNECTION_TIMEOUT but it also includes the time for the initial network connection. Therefore Oracle and CIS recommend setting this value slightly higher than the INBOUND_CONNECTION_TIMEOUT value. By default this parameter is not set. To set this parameter, save a backup of the sqlnet.ora file, add the following line: SQLNET.INBOUND_CONNECTION_TIMEOUT n where n is slightly higher than the value for INBOUND_CONNECTION_TIMEOUT, and then bounce the listener.
24. SQLNET.EXPIRE_TIME is an interesting parameter with multiple purposes. The original purpose was dead connection detection, that is, to detect abnormally terminated client-side connections and terminate the corresponding process on the server side. So if a client kills its connection and this parameter is not set, the server will indefinitely keep the server-side process active thus using valuable server resources for no good purpose. Setting this parameter to 10 will result in a probe being initiated every 10 minutes from the server to the client to see if the client connection has terminated and if so closing the server-side process. The other use for this parameter has developed out of the contrary goals of modern connection pooling (especially for J2EE apps) and firewall technology. Establishing a database connection is a relatively expensive and slow process from an application perspective. Connection pools resolve the problem by keeping open some set minimum number of connections, even when idle, such that connections are normally available to the application. This usually means you'll have at least some idle connections at all times. Enter firewalls, which are generally configured to tear down idle connections (without notifying the client) after some set period of inactivity. When a firewall does this tear down, the client has no idea and assumes it has a good database connection; however, when the pool tries to reuse a torn down connection it get an error. Even if the application is well written and handles the exception, it will still take considerable time to attempt to reuse the connection, get the error, handle the error and retry the connection. If the firewall has torn down multiple connections, which is typically the case, this whole process my be repeated several times for multiple users causing sever performance problems. Setting SQLNET.EXPIRE_TIME causes the probe to be sent from server to client even on idle but active connections. This probe represents activity to the firewall and it therefore no longer considers the connection to be idle and therefore will not tear it down. So if you use connection pools and have a firewall between your app and database tier, set SQLNET.EXPIRE_TIME to no more than half the value of you firewall's idle connection timeout in minutes.
25. Block access to your database listener by IP address. Even if your firewall is compromised, you'll have another layer of security protecting the listener. To enable this feature, backup your sqlnet.ora file, then add the following lines:
    • tcp.validnode_checking = yes
    • tcp.invited_nodes = (hostname1, hostname2, ... , hostnameN)
    • You can specify a list of hostnames or IP addresses. If you use hostnames, they must be resolvable to an IP address or the listener will not start.
    • Unfortunately, the node list does not support partial IP addresses or subnet masks in the manner of an Apache configuration. You must specify a full IP address or a resolvable hostname.
    • You can use tcp.excluded_nodes instead of tcp.invited_nodes; however, the two are mutually exclusive. If you specify both, the excluded nodes will be ignored.
26. 11g comes with a more secure listener out of the box. You can only stop the listener as the local OS account that started the listener. Therefore, it is now actually more secure to NOT set a listener password since setting one enables remote administration. Scanners will likely ding you on this but it is the appropriate practice for 11g.
27. By default, Oracle registers an XPT service. If you are not using DataGurad you should remove this service by running the following as SYS:
    • alter system set "__dg_broker_service_names" = '' scope=spfile;
    • Note the parameter starts with two underscores.
    • Then bounce the database
    • lsnrctl status should now no longer show the XPT service
28. This one is a little off topic but useful to know. If you have an existing database with XML DB installed but you do not need it, you can disable the functionality and reduce your attack vectors by running the following as SYS: alter system reset dispatchers scope=spfile sid='*';
29. Next you'll want to explicitly set the hidden parameter _trace_files_public to false. To do this:
    • create pfile from spfile;
    • Add *._trace_files_public=false to the pfile
    • shutdown immediate
    • startup pfile='<full-path-to-pfile>'
    • create spfile from pfile;
    • shutdown immediate
    • startup
30. Finally, download, read and run the basic CIS Hardening Script. Make sure to read the comments in the script to make sure the settings as appropriate for your environment. The database will need to be bounced for these changes to take effect. This script sets:
    • GLOBAL_NAMES=TRUE
    • OS_AUTHENT_PREFIX TO THE NULL STRING
    • SQL92_SECURITY=TRUE
    • AUDIT_SYS_OPERATIONS=TRUE

         
       

Wall it Off!

Let's talk about direct attacks on the database and the single best way to deter them. Such attacks usually need either a direct user connection to the database server (you would never allow such a thing, would you?) or the ability to get to the database server via one of the many ports Oracle enables for different components. On an Oracle 10g Release 2 database there can be upwards of 20 such ports depending on what components are installed and this does no take into account the ports that are used by the operating system on the database server itself. Each of these ports represents a potential attack vector and the single best way to defend against such an attack is to put the database server behind a firewall, and yes, I mean even from trusted users on your intranet.

Of course you can and should change the default port settings but some are not configurable and anyway this is just a minor bit of slight of hand that will not stop someone determined enough to run a port scan (unless of course you had that firewall).

Sure, you should harden the database listener and configure the invited nodes list to allow connections only from trusted servers (hopefully only application servers and not end user workstations). Great, and what about those other 19 ports??? Most of these are ports only an administrator needs but without a firewall they are there for the hacking. In addition to the listener, there are three for Enterprise Manager, three for iSQL*Plus (please don't tell me you installed this component on a production database), three more for Ultrasearch, two for XML DB and five or six more for RAC depending on the operating system, plus a smattering of others.

Is this iron-clad protection - no! Firewalls can be hacked like anything else and SQL injection attacks from the application tier can cause severe damage without directly attacking the database itself from a client. Still, a firewall is the first and best way to prevent direct attacks on your database.

The next best way is to not allow end-users to directly connect to the database server at all but that will remain a topic for the future.

Security patching can make you LESS secure!

Life was good. Our Oracle database was locked down. Scott and his infamous feline password were long ago dropped. The listener was harder than the Saturday New York Times crossword. Nessus had nothing on me!

Then along came the Oracle CPU. What, you say, our DB version is no longer supported by the "lifetime support" policy? So we upgrade and we patch and we scan again; fully expecting a clean bill of health.

But while we did manage to patch the vulnerabilities, our DB is no longer locked down. Scott has risen from the dead along with his infamous default password. The listener is now softer than the money running Washington.

The moral: Attempts to make your database a little more secure can make it a lot less secure.

Make sure you scan your Oracle DBs regularly but particularly after any upgrades or patches. Invest in a tool. I like AppSentry from Integrigy because I work a lot with the Oracle E-Business Suite where this tool is unmatched. But it works great for plain old Oracle databases as well.  See Integrigy's Web Site  for details and Stephen Kost's Oracle Security Blog at the same site is always a worthy read.