All 10 entries tagged Work

View all 245 entries tagged Work on Warwick Blogs | View entries tagged Work at Technorati | View all 21 images tagged Work

October 12, 2005

Joel on Software – Set Your Priorities

Writing about web page http://www.joelonsoftware.com/articles/SetYourPriorities.html

Joel Spolsky is a clever guy. In case you've not come across him before, he's a genius software engineer who provides some great insight into the software development business.

If you're in IT, it wouldn't hurt to go back and read some of his archives as he is a great communicator with some great ideas.

His latest is all about setting priorities for features. If you're never sure which feature to do next…read it.


LDAP connection pooling

We recently had problems with load on our single sign on (SSO) server. Being the start of term, things are generally busier than the rest of the year and we often see higher load than normal. However, this was too far from normal to be right.

A bit of investigation showed that our JBoss instance had literally 100s and 100s of threads. Lsof is a very handy utility in cases like this.

lsof -p <procid>

This revealed 100s of open connections to our LDAP servers. Not good.

Looking at the LDAP code we have, there are two places where we make LDAP connections, or as they are known in Java; contexts.

Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, 
    "com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.PROVIDER_URL, "ldap://ourldap.warwick.ac.uk");
LdapContext ctx = new InitialLdapContext(env,null);
// do something useful with ctx
ctx.close()

This is pretty much how our code worked in both places. Importantly I'd checked that the contexts were always closed…and they were.

This is where LDAP connection pooling came into the picture. It turned out that one piece of code (not written by us), used this:

env.put("com.sun.jndi.ldap.connect.pool", "true");

This turns on connection pooling. However, we didn't use pooling in the other bit of code. So, one of the other wasn't working. Trying out pooling on both bits of code didn't improve things either, basically because it is a multi–threaded application with 100's of requests a minute, if you just keep creating new LdapContext's from a LdapCtxFactory, you are using a new LdapCtxFactory every time.

Thankfully our SSO application uses Spring so it was simple enough to create an XML entry for the LdapCtxFactory and the environment config and plug the same LdapCtxFactory into the two places it was needed. At least now we were using the same factory.

We could now do this:

Map env = new Hashtable();
env.putAll(getLdapEnv());
env.put("java.naming.security.principal", user);
env.put("java.naming.security.credentials", pass);
LdapContext ldapContext = (LdapContext) getLdapContextFactory().getInitialContext((Hashtable) env);

Where the base LDAP environment and LdapCtxFactory was injected into where it was needed. Then just the username and password to bind as is passed in dynamically.

To really know if pooling is working you need to turn on the debugging for the ldap connection pooling by adding a java option to your test/application/server. There are other handy options for tweaking the pooling behaviour as well.

-Dcom.sun.jndi.ldap.connect.pool.debug=fine
-Dcom.sun.jndi.ldap.connect.pool.initsize=20 -Dcom.sun.jndi.ldap.connect.pool.timeout=10000

The bugging will give you messages like this if pooling isn't working:

Create com.sun.jndi.ldap.LdapClient@c87d32[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@c87d32
Create com.sun.jndi.ldap.LdapClient@c81a32[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@c81a32
Create com.sun.jndi.ldap.LdapClient@a17d35[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@a17d35
Create com.sun.jndi.ldap.LdapClient@1a7e35[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@1a7e35

New connections are just being created every time with no reuse. What you should see is:

Use com.sun.jndi.ldap.LdapClient@17bd5d1
Release com.sun.jndi.ldap.LdapClient@17bd5d1
Create com.sun.jndi.ldap.LdapClient@cce3fe[nds.warwick.ac.uk:389]
Use com.sun.jndi.ldap.LdapClient@cce3fe
Release com.sun.jndi.ldap.LdapClient@cce3fe
Use com.sun.jndi.ldap.LdapClient@1922b38
Release com.sun.jndi.ldap.LdapClient@1922b38
Use com.sun.jndi.ldap.LdapClient@17bd5d1
Release com.sun.jndi.ldap.LdapClient@17bd5d1

As you can see, there are actually two differences here from a fully working connection pool and a well and truely broken one.

  1. There are very few creates and lots of reuse in the good code
  2. There are lots of releases after connection use in the good code

This is where we came across our second problem. Although in theory the connection pooling was working and I could see some reuse, it was still creating a lot of connections and I wasn't seeing barely any 'Release' messages.

Chris hit the nail on the head with pointing out that NamingEnumerations could well be just like PreparedStatements and ResultSets for JDBC. It is all fine and well closing the connection/context itself, but if you don't close the other resources, it won't actually get released.

The proof of this shows up again in lsof or netstat. A context that has been closed but still has an open NamingEnumeration shows up like this:

java    21533 jboss   80u  IPv6 0x32376e2cf70   0t70743    TCP ssoserver:60465->ldapserver.warwick.ac.uk:ldap (ESTABLISHED)

However, when it is closed, it should wait to be closed, like this:

java    21533 jboss   80u  IPv6 0x32376e2cf70   0t70743    TCP ssoserver:60465->ldapserver.warwick.ac.uk:ldap (TIME_WAIT)

Upon closing all NamingEnumerations, we finally got the perfect results. 100s of requests a minute and only ever around 10–15 ldap connections open at any one time.

So, lessons learnt.

  • When creating contexts, share the factory to use pooling
  • Make sure you close everything. If it has a close()...use it!
  • Occasionally take a look at the open connections and threads that you application has…it might surprise you.

Update:

Spring config:


<bean id="ldapContextFactory" class="com.sun.jndi.ldap.LdapCtxFactory" singleton="true"/>

<bean id="ldapEnv" class="java.util.Hashtable">
<constructor-arg>
<map>
<entry key="java.naming.factory.initial"><value>com.sun.jndi.ldap.LdapCtxFactory</value></entry>
<entry key="java.naming.provider.url"><value>ldaps://ourldap.ac.uk</value></entry>
<entry key="java.naming.ldap.derefAliases"><value>never</value></entry>
<entry key="com.sun.jndi.ldap.connect.timeout"><value>5000</value></entry>
<entry key="java.naming.ldap.version"><value>3</value></entry>
<entry key="com.sun.jndi.ldap.connect.pool"><value>true</value></entry>
        </map>
</constructor-arg>
</bean>

Update:
We now do connection pooling with LDAPS so we use the additional system property:

-Dcom.sun.jndi.ldap.connect.pool.protocol="plain ssl"

October 06, 2005

Bulk deleting bad data

I had to clear up some old bad data that was left over from a bit of bad code. Unfortunately the bad data didn't rear its ugly head until recently and a lot of data had built up. It was also very hard to detect the bad data because of many places it could be referenced from…only if it had no references to it from any of 7 places would it need to be deleted.

This means doing a really horrible query either like this:

select id from atable where
id not in (select id from anothertable)
and
id not in (select id from yetanothertable)
and
id not in (select id from moretables)
.....
.....

This is very, very, very slow.

A more efficient way of doing this is this:

select id from atable a where
not exists (select id from anothertable b where a.id = b.id)
and
not exists (select id from yetanothertable c where a.id = c.id)
and
not exists (select id from moretables d where a.id = d.id)
.....
.....

However, when you are dealing with potentially 100,000's of rows it is still quite slow…but it does get there. The next problem is actually deleting the data once you've managed to select it. As a little test I thought I'd try and delete the whole lot, but that just didn't work…too slow. Even if I did have the patience to leave it running for hours, I couldn't let it lock up the database like that for that long.

So, the only solution was to do it in batches. I wrote a quick java program that would iterate through an do the deletes in small batches of a 100 or so at a time. My first mistake was trying to reuse some Spring/Hibernate code I already had instead of going straight to old school and using JDBC.

Although in theory you can get a Connection object from the hibernate Session, via session.connection(), it really is NOT the same as just getting a good old fashioned JDBC connection. The deletes were taking absolutely ages, so I profiled it and noticed that hibernate was still trying to do some of its funky stuff in the background, really slowing things down.

Plan B (or is it D by now?). Spring comes with a handy little JdbcTemplate which lets you do real JDBC but without a lot of the exception and connection/statement/resultset closing pains. Finally…it worked.

So, lesson of the day:

  • not exists type queries are faster than not in queries
  • Bulk deletes can be verrrrrrry slow
  • Batching deletes is better, but with real JDBC not hibernate SQL calls

October 04, 2005

Parking warning

I got a note on my car today warning me that they are going to start fining and clamping so if you're going to park somewhere dodgy, think twice tomorrow.

September 30, 2005

Cautionary tale

Writing about web page http://www.kuro5hin.org/story/2005/9/27/95759/4240

Just read a scary story about the power of big consultancy firms to make huge amounts of money from rich and ignorant companies/managers.


At this point, IBM Global Services consultants flooded our conference rooms. Overnight, we ended up with twenty consultants. When I asked how much these consultants were costing us, I was told $250/hr. This information proved to be incorrect – they were actually charging us $325/hr.

What were we getting for $325/hr? People hired off of Monster and Careerbuilder. Seriously.

Management was under the assumption that we would be getting real implementation experts from IBM. In fact, we were getting employees from a subcontractor. We paid IBM $325/hr, and they paid their subcontractor about $165/hr. The subcontractor then paid its people salaries of $90,000 to $110,000/yr, the market average, which equates to about $75/hr when benefits are included. We were paying a markup of about 333%.

It's a depressing tale…but not if you were one of the consultants.


September 27, 2005

A year of Warwick Blogs

So, Warwick Blogs has been officially live for a year now. We've certainly come a long way in 12 months. It's almost hard to remember what it felt like at the start of last year, but this year certainly feels calmer as Warwick Blogs is no longer the new exciting thing it used to be.

So, how has it weathered once the initial interest died down? Depends who you ask and how you measure success, but I would (of course I would) say that it's been a success.

We've certainly let potentially thousands of staff and students express themselves online in a way that they just couldn't before and out of that has come some 42,000 entries, over 100,000 comments and 53,000 images. In the past there was just no easy way to share your thoughts, ideas, rants and photos with the Warwick community (and beyond).

When asked at the start of last academic year how many blogs and entries I thought we would get by Xmas, I said that I really had no idea what so ever as what we were trying was so untested within a university in this country and almost anywhere else. By the end of November 2004 we were getting 20,000 page views per day (proper page views not including images/robots/etc.). As it turned out we had 13,000 entries, 2,000 blogs and 26,000 comments at Xmas, turning it into the biggest educational blogging site in the world in just 3 months.

We are clearly not going to see that kind of rocketing growth again this year as it is only new to the 1st years rather than the whole University this time. However, we have steadily increased the features, reliability and power of the system over the last year and hope that there are new things to offer people who didn't try out blogs before or who gave up early on. Our BlogBuilder News blog has kept track of these changes and shows how far we've come.

I like my graphs, so here are a few showing a years worth of Warwick Blogs (they look a bit poor, but click on them to see a larger view of each graph)

The above entries per day graph shows that we really do quite closely follow the term and holiday times with a hardcode of bloggers keeping going over the holidays. Generally speaking though we've stood at around an average of 150 new entries per day. I must admit a little disappointment that this has not grown much more, but it is still more than I had originally hoped for.

Comments per day is an interesting one. Again it tracks term time very well, but pleasingly it also shows that we have an average of 2/3 comments per entry, showing a good community around Warwick Blogs as the average entry on a blog in the wide world will be lucky to every get a comment at all.

Images were another nice surprise for us. Although at the start we only provided single image upload (now we have zip upload and nicer galleries), people were uploading pictures left, right and centre. We have more images than we do entries, showing that people love to share their pictures, I can only hope that that will continue this year. Now that we have image tagging (like Flickr), finding and categorising images across the system is even easier.

This is a slightly strange one in the sense that the growth is fairly clear, whereas the number of new comments/entries/images a day has not really kept growing, page views has. These stats are for real people viewing pages of entries, not images/css/javascript/robots/etc. We had 30,000 page views yesterday for instance.

Karen and Hannah have yet again done some great publicity again this year, but the emphasis has been a little more on the academic side of using blogs this time as we didn't really push it that way last year. Fingers crossed we can get some more involvement from our academics and get them communicating and interacting with their students online a bit more.


September 08, 2005

Out of office blog entry

After a long old wait, I am finally off to South Africa to visit my parents. My next 24 hours consists of:

  • My taxi will be here in half an hour
  • Coach at 7:00 from Coventry
  • 09:30 arrive at Heathrow
  • 10:00 meet my brother
  • 11:00 check in
  • 13:50 fly to Zurich
  • 16:35 land in Zurich
  • Kill 6 hours!
  • 10:20 fly out of Zurich
  • Sleeeeeeeeep
  • 08:35 land in Johanesburg
  • Meet parents and start holiday!

ta ta…


September 06, 2005

Firefox developers plugins

Writing about web page http://www.bytefarmers.com/log/2005/09/03/extending-firefox-for-web-developers/

To add to my usual collection of Firefox plugins, I came across the above webpage listing a few new ones. On top of my existing ones, I've just added:

View Rendered Source by Jennifer Madden
Neat tool that allows to view the actual generated markup, useful when generating markup via scripting.

View Cookies by Edwin Martin
Utility that adds a Cookie tab on the ‘Page Info’ window, allowing you to see and edit any set cookie that affects that page.

ViewCookies is so much quicker and easier than browsing around the standard Firefox cookies menu and the View Rendered Source is lovely for viewing HTML source code.


September 02, 2005

External Spring configuration

I recently had to generalise some configuration I had for Single Sign On (SSO). Some configuration which I didn't think was going to change on a per server/deploy basis, does indeed need to change. Generally we don't like to do a different build per different server so we try to put settings that will change on a per server basis into external configuration.

Traditionally this is done via system properties (System.setProperty("sso.someproperty","true");) which we set in the properties-service.xml file in the JBoss deploy directory. These are really handy as they are dynamically reloaded every time you change them so you can quickly change something at run time. We use this for hot changing where web services point at for instance in case one breaks.

However, there are sometimes bigger configuration changes that you might want to make which are not really just single strings. Not only that, the system properties don't really fit in very well with a Spring way of thinking.

Spring offers two possibilities. From their documentation they talk about the PropertyPlaceholderConfigurer but that just puts individual string properties in an external properties file again.

I decided just to split out the parts of my XML bean configuration that are per server specific into separate file. You can then just dump a smaller configuration file somewhere in your classpath (such as the JBoss conf directory) that is per server specific. In your web.xml you can reference your configuration like this:

<context-param>
   <param-name>contextConfigLocation</param-name>
   <param-value>/WEB-INF/main-spring-beans.xml,classpath:deploy-specific-beans.xml</param-value>
</context-param>

And there you go, you have a nice Spring XML configuration to put your server specific configuration in, without a separate build.


Cubicle protection!

Writing about web page http://www.thinkgeek.com

We will be going through a possible little office cubicle shuffle soon and I overheard people talking about cubicle privacy and it reminded me of these:

Room defender

Lazer Trip Wire

Defend your cubicle!


February 2023

Mo Tu We Th Fr Sa Su
Jan |  Today  |
      1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28               

Tags

Search this blog

Most recent comments

  • One thing that was glossed over is that if you use Spring, there is a filter you can put in your XML… by Mathew Mannion on this entry
  • You are my hero. by Mathew Mannion on this entry
  • And may all your chickens come home to roost – in a nice fluffy organic, non–supermarket farmed kind… by Julie Moreton on this entry
  • Good luck I hope that you enjoy the new job! by on this entry
  • Good luck Kieran. :) by on this entry

Galleries

Not signed in
Sign in

Powered by BlogBuilder
© MMXXIII