October 12, 2005

LDAP connection pooling

We recently had problems with load on our single sign on (SSO) server. Being the start of term, things are generally busier than the rest of the year and we often see higher load than normal. However, this was too far from normal to be right.

A bit of investigation showed that our JBoss instance had literally 100s and 100s of threads. Lsof is a very handy utility in cases like this.

lsof -p <procid>

This revealed 100s of open connections to our LDAP servers. Not good.

Looking at the LDAP code we have, there are two places where we make LDAP connections, or as they are known in Java; contexts.

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, 
    "com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.PROVIDER_URL, "ldap://ourldap.warwick.ac.uk");
LdapContext ctx = new InitialLdapContext(env,null);
// do something useful with ctx
ctx.close()

This is pretty much how our code worked in both places. Importantly I'd checked that the contexts were always closed…and they were.

This is where LDAP connection pooling came into the picture. It turned out that one piece of code (not written by us), used this:

env.put("com.sun.jndi.ldap.connect.pool", "true");

This turns on connection pooling. However, we didn't use pooling in the other bit of code. So, one of the other wasn't working. Trying out pooling on both bits of code didn't improve things either, basically because it is a multi–threaded application with 100's of requests a minute, if you just keep creating new LdapContext's from a LdapCtxFactory, you are using a new LdapCtxFactory every time.

Thankfully our SSO application uses Spring so it was simple enough to create an XML entry for the LdapCtxFactory and the environment config and plug the same LdapCtxFactory into the two places it was needed. At least now we were using the same factory.

We could now do this:

Map env = new Hashtable();
env.putAll(getLdapEnv());
env.put("java.naming.security.principal", user);
env.put("java.naming.security.credentials", pass);
LdapContext ldapContext = (LdapContext) getLdapContextFactory().getInitialContext((Hashtable) env);

Where the base LDAP environment and LdapCtxFactory was injected into where it was needed. Then just the username and password to bind as is passed in dynamically.

To really know if pooling is working you need to turn on the debugging for the ldap connection pooling by adding a java option to your test/application/server. There are other handy options for tweaking the pooling behaviour as well.

-Dcom.sun.jndi.ldap.connect.pool.debug=fine
-Dcom.sun.jndi.ldap.connect.pool.initsize=20 -Dcom.sun.jndi.ldap.connect.pool.timeout=10000

The bugging will give you messages like this if pooling isn't working:

Create com.sun.jndi.ldap.LdapClient@c87d32[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@c87d32
Create com.sun.jndi.ldap.LdapClient@c81a32[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@c81a32
Create com.sun.jndi.ldap.LdapClient@a17d35[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@a17d35
Create com.sun.jndi.ldap.LdapClient@1a7e35[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@1a7e35

New connections are just being created every time with no reuse. What you should see is:

Use com.sun.jndi.ldap.LdapClient@17bd5d1
Release com.sun.jndi.ldap.LdapClient@17bd5d1
Create com.sun.jndi.ldap.LdapClient@cce3fe[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@cce3fe
Release com.sun.jndi.ldap.LdapClient@cce3fe
Use com.sun.jndi.ldap.LdapClient@1922b38
Release com.sun.jndi.ldap.LdapClient@1922b38
Use com.sun.jndi.ldap.LdapClient@17bd5d1
Release com.sun.jndi.ldap.LdapClient@17bd5d1

As you can see, there are actually two differences here from a fully working connection pool and a well and truely broken one.

  1. There are very few creates and lots of reuse in the good code
  2. There are lots of releases after connection use in the good code

This is where we came across our second problem. Although in theory the connection pooling was working and I could see some reuse, it was still creating a lot of connections and I wasn't seeing barely any 'Release' messages.

Chris hit the nail on the head with pointing out that NamingEnumerations could well be just like PreparedStatements and ResultSets for JDBC. It is all fine and well closing the connection/context itself, but if you don't close the other resources, it won't actually get released.

The proof of this shows up again in lsof or netstat. A context that has been closed but still has an open NamingEnumeration shows up like this:

java    21533 jboss   80u  IPv6 0x32376e2cf70   0t70743    TCP ssoserver:60465->ldapserver.warwick.ac.uk:ldap (ESTABLISHED)

However, when it is closed, it should wait to be closed, like this:

java    21533 jboss   80u  IPv6 0x32376e2cf70   0t70743    TCP ssoserver:60465->ldapserver.warwick.ac.uk:ldap (TIME_WAIT)

Upon closing all NamingEnumerations, we finally got the perfect results. 100s of requests a minute and only ever around 10–15 ldap connections open at any one time.

So, lessons learnt.

  • When creating contexts, share the factory to use pooling
  • Make sure you close everything. If it has a close()...use it!
  • Occasionally take a look at the open connections and threads that you application has…it might surprise you.

Update:

Spring config:


<bean id="ldapContextFactory" class="com.sun.jndi.ldap.LdapCtxFactory" singleton="true"/>

<bean id="ldapEnv" class="java.util.Hashtable">
<constructor-arg>
<map>
<entry key="java.naming.factory.initial"><value>com.sun.jndi.ldap.LdapCtxFactory</value></entry>
<entry key="java.naming.provider.url"><value>ldaps://ourldap.ac.uk</value></entry>
<entry key="java.naming.ldap.derefAliases"><value>never</value></entry>
<entry key="com.sun.jndi.ldap.connect.timeout"><value>5000</value></entry>
<entry key="java.naming.ldap.version"><value>3</value></entry>
<entry key="com.sun.jndi.ldap.connect.pool"><value>true</value></entry>
        </map>
</constructor-arg>
</bean>

Update:
We now do connection pooling with LDAPS so we use the additional system property:

-Dcom.sun.jndi.ldap.connect.pool.protocol="plain ssl"

- 16 comments by 1 or more people Not publicly viewable

[Skip to the latest comment]
  1. Nick Faiz

    Great post – you saved me much time in debugging some performance issues in an app. I was building.

    Cheers,
    Nick

    03 Nov 2005, 21:59

  2. Cool post.

    Never used LDAP, but handy to know…..

    08 Nov 2005, 17:06

  3. Derek Morr

    Thanks for the post! Very interesting.

    09 Nov 2005, 13:20

  4. Dmitry Brin

    One thing that I found is that the Debug and Timeout option can not be set through env properties. They must be specified though java command option -D…. .

    01 Dec 2005, 03:12

  5. Phil

    Cheers – had exactly the same issues, and saved us a lot of time!

    05 Dec 2005, 01:47

  6. Alex

    I think the idea for pooling is to reuse the TCP connection, and only do rebinding with different user/password when the context is reused from the pool. So you shouldn't expect the underlying TCP socket to be closed, even when the context is released back to the pool.

    >
    > However, when it is closed, it should wait to be closed, like this:
    > java 21533 jboss 80u IPv6 0×32376e2cf70 0t70743 TCP ssoserver:60465->ldapserver.warwick.ac.uk:ldap (TIME_WAIT)

    Does that make sense?

    -Alex

    21 Dec 2005, 20:51

  7. Alex

    A correction to my previous comment. Sun's ldap pooling doesn't support reuse of different principle/credential bindings. Though the conclusion is still true. -Alex

    21 Dec 2005, 21:59

  8. siwecki

    When you get results form ldap as objects, you should also close them. Example:

    SearchControls sc = new SearchControls();
    //.. set your searchControls..
    sc.setReturningObjFlag (true);

    NamingEnumeration result = ctx.search("DC=mycompany,DC=com", "userPrincipalName=p*", sc);

    while (result.hasMoreElements ())
    {

    SearchResult sr = ((SearchResult) result.nextElement());

    LdapCtx obj2 = (LdapCtx) sr.getObject();

    //do sth with obj2

    //line below is important

    obj2.close();
    }

    //also you must close result and context
    result.close;
    ctx.close;

    30 Mar 2006, 11:12

  9. Dolan

    Fantastic writeup – helped me a lot. Would be great if you could include the spring config for the ldap env and context factory.

    09 May 2006, 22:59

  10. mark

    Has anyone attempted to simultaneously pool connections to two different LDAP providers? The support SEEMS to limit me to a single provider. I need to use one directory to fetch information from another (completely separate) directory. If I try to pool connections to both, things get stepped on.

    07 Jun 2006, 15:22

  11. How do you mean things get stepped on? Are you using a single LdapCtxFactory or one for each directory?

    07 Jun 2006, 15:26

  12. sagar

    I close all NamingEnumerations and Contexts in my code but still have this issue of connections not being closed. Can som eone help me in this ?

    24 Jun 2006, 01:06

  13. serhat

    hi

    is there any way to authenticate different users by using ldap connection pooling.as Alex said ,Sun's API does not support reuse of different principle/credential bindings.I didn't understand how username and password is passed in dynamically to env.i wrote a bit of code but it didnt work fine.

    regards

    20 Jul 2006, 13:10

  14. Map env = new Hashtable();
    env.putAll(getLdapEnv());
    env.put("java.naming.security.principal", user);
    env.put("java.naming.security.credentials", pass);
    LdapContext ldapContext = (LdapContext) getLdapContextFactory().getInitialContext((Hashtable) env);

    Here your main LDAP env has most of your connection settings, but then you just dynamically add the principal and credentials attributes when you get the actual connection

    21 Jul 2006, 09:04

  15. Suresh

    Hi,

    The article has been extremely useful. We have one other issue with connection pooling, for which help would be appreciated.

    We are using connection pooling on JBoss and find that if a user changes their password, the old password is accepted for a few minutes since the change. Is there a way to flush connections for which credentials have been changed?

    Regards

    21 Aug 2006, 15:48

  16. Hi Suresh,

    That's a good point and in fact one that we didn't come across until recently. We only recently started doing password changes from within our application so in general password changes didn't happen very often and were not a problem. Now however we do this:

    public LdapContext connectToLdap(final String user, final String pass, final LdapCtxFactory ldapCtxFactory,
    final Map ldapEnv, boolean doPooling) throws NamingException {

    Map env = new Hashtable();
    env.putAll(ldapEnv);
    if (doPooling) {
    env.put("com.sun.jndi.ldap.connect.pool", "true");
    } else {
    env.put("com.sun.jndi.ldap.connect.pool", "false");
    }
    env.put("java.naming.security.principal", user);
    env.put("java.naming.security.credentials", pass);
    LdapContext ldapContext = (LdapContext) ldapCtxFactory.getInitialContext((Hashtable) env);

    LOGGER.debug("Got LDAP connection:" + ldapContext.toString());

    return ldapContext;

    }
    Which is basically just pass in a flag that lets us choose if we use pooling for this connection or not. So when we are doing end user authentication connections we do not pool, but when we are doing a search of some kind (our most common use case) with an admin user that has browse rights then we do use pooling. For us end user authentication does not have to be that quick as it doesn't happen that often compared to user searches

    21 Aug 2006, 17:09


Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

October 2005

Mo Tu We Th Fr Sa Su
Sep |  Today  | Nov
               1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31                  

Tags

Search this blog

Most recent comments

  • One thing that was glossed over is that if you use Spring, there is a filter you can put in your XML… by Mathew Mannion on this entry
  • You are my hero. by Mathew Mannion on this entry
  • And may all your chickens come home to roost – in a nice fluffy organic, non–supermarket farmed kind… by Julie Moreton on this entry
  • Good luck I hope that you enjoy the new job! by on this entry
  • Good luck Kieran. :) by on this entry

Galleries

Not signed in
Sign in

Powered by BlogBuilder
© MMXIX