Favourite blogs for Jeff's Irregular Movements

My favourites » All entries

April 12, 2018

Using Bootstrap

Writing about web page https://johndale.info

Learning how to use Bootstrap cards to make a responsive website – https://johndale.info. Fun!

May 01, 2014

memory_target sga_target and db_cache_size

On my two databases, the buffer cache were way too small. One buffer was 128 mb, which was the default value set in db_cache_size; one is 64 mb while the db_cache_size value is 0.

On both servers, the memory_target parameters were set (5g and 3g respectively) while sga_targets are set to zero. It seemed ASSM mis-configured the buffer cache.

After I set the sga_target to a reasonable value, ASSM began to adjust the cache buffer and set them to a much higher value. The first one was set to 700 mb and the second 608 mb.

Both databases are on solaris 10.

Maybe we should either set the sga_target, or set a high value in db_cache_size.

February 23, 2012

Instantiate table in Oracle Streams

Writing about web page http://krish-dba.blogspot.com/2009/01/re-synchronizingrefresh-table-in.html

Credit to this article

1. Stop Capture

2. Stop propagation

3. Stop Apply Process on Target(T)


4. Delete the apply errors on Target

Check the errors in the error queue:

SQL> select apply_name, LOCAL_TRANSACTION_ID, ERROR_MESSAGE from dba_apply_error;

To delete specific transaction:


To delete all errors: 


5. Truncate table 

Truncate table UWTABS.UW_TPRS

6. Re-instantiate the problem table on Source(S)


 source_object_name => 'UWTABS.UW_TPRS', 
 source_database_name => 'source.warwick.ac.uk', 
 instantiation_scn => iscn);

Synchronize the problem table through datapump exp/imp

8. Start the Apply Process on Target(T)


9. Start the propagation Process on the Source(S)


10. Start the Capture Process on the Source(S)


February 21, 2012

job name in datapump

Pay attention to the format of job name

oracle@db$ expdp / attach=nightly_pump_210212-1059

ed to: Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning and Real Application Testing options
ORA-31626: job does not exist
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.KUPV$FT", line 405
ORA-31638: cannot attach to job nightly_pump_210212-1059 for user NIGHTLY_PUMP_210212
ORA-31632: master table "NIGHTLY_PUMP_210212.nightly_pump_210212-1059" not found, invalid, or inaccessible
ORA-00942: table or view does not exist

Datapump can find correct user after changing the job name.

oracle@db$ expdp / attach=nightly_pump_210212_1059
Connected to: Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Partitioning and Real Application Testing options
ORA-31626: job does not exist
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.KUPV$FT", line 405
ORA-31638: cannot attach to job NIGHTLY_PUMP_210212_1059 for user OPS$ORACLE
ORA-31632: master table "OPS$ORACLE.NIGHTLY_PUMP_210212_1059" not found, invalid, or inaccessible
ORA-00942: table or view does not exist

Seemed the ”-” in the job name caused problem.

January 25, 2012

Use Timestamp type instead of Date type

Today I noticed another problem caused by Date type.

select page3_.*
from sb2.NEWS_ITEM this_ inner join sb2.CONTENT_FETCHER cf1_ on this_.news_id=cf1_.ID
 inner join sb2.CONTENT content2_ on cf1_.content_id=content2_.id 
  inner join sb2.PAGE page3_ on content2_.page_id=page3_.id
where ( this_.discriminator = 'calendar_item') and page3_.LAST_UPDATED_DATE>:1 
order by page3_.LAST_UPDATED_DATE asc, this_.publication_date asc

The SQL Tunning Advisor suggested that “page3_.LAST_UPDATED_DATE>:1 ” has an implicit data type conversion prevent the optimizer from selecting indices.

The LAST_UPDATED_DATE column is of “DATE” type.

From V$SQL_MONITOR, the bind variable is:
<bind name=":1" pos="1" dty="180" dtystr="TIMESTAMP" maxlen="11" len="7" format="hexdump">78640101010101</bind>

After inspecting the source code, it is appear bind as java.util.Date.

I think it should be bound as java.sql.Date, not java.sql.Timestamp or java.util.Date. See http://databaseperformance.blogspot.com/2009/01/java-data-types-and-oracle-index-usage.html

Oracle saw the value it was getting was of type TIMESTAMP, which has sub-second values in it. And rather than lose precision by truncating its value, it was instead converting the DATE value in each data row to a TIMESTAMP value before comparing.

Better to use Timestamp type instead of Date type. The index should work properly whether you bind java.sql.Date or java.sql.Timestamp.

Data Type Conversion
Oracle will convert the value to most precise type:

Column type DATE, predicate value is TIMESTAMP => Right size is more precise => Convert every row to timestamp
Column type Timestap, predicate value is DATE => Left size is more precise => convert predicate to Timestap
Column type varchar, predicate value is number => Right size is more precise => convert every row to varchar
Column type is number, predicate value is varchar => Left size is more precise => convert predicate value to number

January 12, 2012

Delete old rman backup piece manually

My Nagios checking reported there were an old backup dated back 2010. There is no such backup on disk. RMAN crosscheck and delete wont be able to remove it.

Tried to delete it with “nice” rman command:
RMAN> delete backup tag = 'TAG20101029T101009';

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=317 device type=DISK
specification does not match any backup in the repository

RMAN> delete backuppiece tag = 'TAG20101029T101009';
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 01/12/2012 09:59:49
RMAN-06168: no backup pieces with this tag found: TAG20101029T101009
Resort to manually deleting in the base table:
SELECT * FROM rc_backup_set where db_id='1320220071' order by start_time
select * from bp where db_key=10492312 and tag = 'TAG20101029T101009'
delete from  bp where db_key=10492312 and tag = 'TAG20101029T101009'

January 10, 2012

Install Oracle Instant Client on Ubuntu

1. Download instantclient-basic-linux.x64- and instantclient-sqlplus-linux.x64- and unzip them into /apps/oracle/instantclient_11_2

2. sudo apt-get install libaio1

3. Append the following line into ~/.bashrc

export LD_LIBRARY_PATH=/apps/oracle/instantclient_11_2
export SQLPATH=~/code/oracle-scripts/scripts:/app/oracle/instantclient_11_2
export TNS_ADMIN=/apps/oracle/instantclient_11_2

No need the TNS_ADMIN if you just want to use easy connect.

4. (optional)
create tnsnames.ora

5. Now sqlplus should works.
sqlplus system/password@oracle_server

This doc help me a lot.

Use online redefinition to change a table into a partitioned table

Created a test user hsun

drop table log purge;
drop table children purge;
drop table parent purge;
drop view children_less_10_v;

create table parent (id number);
alter table parent add constraint pk_parent_id primary key (id);

create table log (msg varchar2(50));

create table children (id number,pid number,  constraint  pk_id  primary key  (id));
alter table children add constraint fk_pid foreign key (pid) references parent (id);
create view children_less_10_v as select * from children where id < 10;

create or replace trigger children_tri 
before insert or update on children 
for each row 
  insert into log values('new' || :new.id);

insert into parent  select rownum from dual connect by level < 15;
insert into children  select rownum, rownum from dual connect by level < 15;

Run the script status.sql that contains the following SQL:

set lin 200;
col object_name format a20
select object_name, object_type, status from user_objects order by 2, 1 ;
select trigger_name , table_name from user_triggers order by 2;
select view_name, text from user_views;
select CONSTRAINT_NAME, table_name, status from user_constraints order by 2;
select table_name, PARTITION_NAME from user_tab_partitions order by 1;

SQL> @/tmp/status.sql
-------------------- ------------------- -------
PK_ID                INDEX               VALID
PK_PARENT_ID         INDEX               VALID
CHILDREN             TABLE               VALID
LOG                  TABLE               VALID
PARENT               TABLE               VALID
CHILDREN_TRI         TRIGGER             VALID
CHILDREN_LESS_10_V   VIEW                VALID

7 rows selected.

TRIGGER_NAME                   TABLE_NAME
------------------------------ ------------------------------
CHILDREN_TRI                   CHILDREN

VIEW_NAME                      TEXT
------------------------------ --------------------------------------------------------------------------------
CHILDREN_LESS_10_V             select "ID","PID" from children where id < 10

CONSTRAINT_NAME                TABLE_NAME                     STATUS
------------------------------ ------------------------------ --------
FK_PID                         CHILDREN                       ENABLED
PK_ID                          CHILDREN                       ENABLED
PK_PARENT_ID                   PARENT                         ENABLED

no rows selected

Run as SYSDBA in a different session:

-- -- Check table can be redefined

drop table hsun.children_p purge;
create table hsun.children_p (id number,pid number) 
 partition by range (id)
 ( partition p1 values less than (5),  
   partition p2 values less than (10),   
   partition p3 values less than (maxvalue));

-- Start Redefinition

  l_num_errors PLS_INTEGER;
       orig_table => 'CHILDREN',
       int_table => 'CHILDREN_P', 
       copy_indexes  => DBMS_REDEFINITION.cons_orig_params,
       num_errors => l_num_errors);
 DBMS_OUTPUT.put_line('l_num_errors=' || l_num_errors);

-- Optionally synchronize new table with interim data before index creation

-- Complete redefinition

drop table hsun.children_p purge;

Run status.sql again:

SQL> @/tmp/sta

-------------------- ------------------- -------
PK_ID                INDEX               VALID
PK_PARENT_ID         INDEX               VALID
CHILDREN             TABLE               VALID
LOG                  TABLE               VALID
PARENT               TABLE               VALID
CHILDREN_LESS_10_V   VIEW                INVALID

10 rows selected.

TRIGGER_NAME                   TABLE_NAME
------------------------------ ------------------------------
CHILDREN_TRI                   CHILDREN

VIEW_NAME                      TEXT
------------------------------ --------------------------------------------------------------------------------
CHILDREN_LESS_10_V             select "ID","PID" from children where id < 10

CONSTRAINT_NAME                TABLE_NAME                     STATUS
------------------------------ ------------------------------ --------
FK_PID                         CHILDREN                       ENABLED
PK_ID                          CHILDREN                       ENABLED
PK_PARENT_ID                   PARENT                         ENABLED

TABLE_NAME                     PARTITION_NAME
------------------------------ ------------------------------
CHILDREN                       P1
CHILDREN                       P3
CHILDREN                       P2

The Database versions

SQL> select * from v$version;

Oracle Database 11g Enterprise Edition Release - 64bit Production
PL/SQL Release - Production
CORE      Production
TNS for Solaris: Version - Production
NLSRTL Version - Production
  • The primary key, foreign key and triggers remains “ENABLED”
  • View and trigger are “INVALID”

January 05, 2012

Solaris df reported disk usage differently to du

I dropped the hitlogging tablespace whose size was 200+ gb.

run df report

bash-3.00# df -h
la-ora01-db            588G    31K    44G     1%    /oradb
la-ora01-db/data       588G   489G    44G    92%    /oradb/data

bash-3.00# du -s -h /oradb/data
 217G   /oradb/data

-bash-3.00# for pid in `ps -ef | grep ora_ | awk '{print $2}'`; do 
 pfiles  $pid | grep log  ; done | sort | uniq
The files I drooped were hitlogging*.dbf and non of them appear in the list above.
After 3-4 hours waiting to remove the possibility of cache effect, I restarted the database

SQL>shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

-bash-3.00# df -h
Filesystem             size   used  avail capacity  Mounted on
/                       39G   8.8G    30G    23%    /
/dev                    39G   8.8G    30G    23%    /dev

la-ora01-db            588G    31K   316G     1%    /oradb
la-ora01-db/data       588G   217G   316G    41%    /oradb/data

Maybe some other user process hold the files? I only inspected the oracle background process and usually the most likely suspect is db file writer (dbwn).

December 31, 2011

Hollyzone's Albums Of 2011 1–10

And here’s the top 10! These are the ten albums I would take if I could only have ten from this year, although why anyone would be so cruel as to only allow me ten albums from any given year is beyond me…

1. Anna Calvi – Anna Calvi

It’s actually illegal in most countries for the under-18s to listen to albums are sexeh as this one, so be grateful we live in an enlightened nation where a woman cooing/bellowing about desire/the devil over Spanish-influenced guitars and occasional bouts of lush strings is totally acceptable. Oh yeah, this is bloody good. It’s all lyrical gender swapping, and bursts of thrilling sound. Heck, it starts with a guitar solo without making you want to frisbee the record right out of the window.
Best bits:
- ‘Suzanne And I’ – everyone loves big, ambiguous anthems about lesbonics/friendship/who knows.
- ‘Desire’ – ecstatic chorus alert.
- ‘Love Won’t Be Leaving’ – kitchen sink outro of joy.

2 Austra – Feel It Break

Do you like a bit of chilly 80s-style synth music? How about it with an ex-opera singer, ex-punk, ex-singer-songwriter in charge? Yes, that is a good plan. That is Austra’s plan. Lobbing lovely glacial synths onto a live rhythm section gives the songs quite a lot of heft and crunch, so even if the album’s a little one paced it’s a good pace, like the march of an army of sad robot penguins. There’s also the matter of Katie Stelmanis’s vocals – it’s pretty standard in chilly synth pop world to have singers who can’t really sing, but she can. So it’s all dizzying harmonies, wordless pizzicato bleeping and big stentorian bellowing, although thankfully none of that diva, singing-more-notes-than-is-necessary shit. Marvellous stuff.
Best bits:
- ‘Darken Her Horse’ – the best ‘big soaring chorus’ of the year, hands down.
- ‘Lose It’ – sounds a little like those Lloyds adverts but is really really good.
- ‘The Beat And The Pulse’ – juddering, Knife-ish, chug, one of the singles of the year.

3 Suuns – Zeroes QC

An ominous drone, a sinister groove, vocals drifting in on a distorted yet still human-sounding wind. Who are Suuns and what do they with our computers and guitars? They’re the twisted, sinister end point for all that indie-electro-dance stuff which has been around in varying degrees of palatability for the last few years, and they have made a damn fine album which isn’t as hard to listen to as it might seem at first. Its ideas are kept brief and breezy, nothing outstays its welcome. It invites us to dance, but in the style of a close second place in the Turing test – these robots have almost made a record which sounds human, but not quite. And that’s why it’s so good.
Best bits:
- ‘Arena’ – a real grower, in that it grows from nothing in a danceable number.
- ‘Up Past The Nursery’ – sparse but still funky, quite a feat in itself.
- ‘Armed For Peace’ – the only track serves as a pretty much perfect introduction to the rest of the album in just three and a half minutes.

4 Gang Gang Dance – Eye Contact

Gang Gang Dance are one of those special bands who can release an album where the first track is eleven minutes long and mostly ambient noise, and yet the album is still their most poppy. There’s a duet with him from Hot Chip, which sounds like a sad nerd serenading a crazy witch. Y’see, Lizzi Bougatsos has a voice like a tiny, eight year old serial killer, which makes it doubly unnerving when she starts singing lyrics nicked from nursery rhymes or ditties about “mindkillas”. It helps that all the tracks are brilliantly buzzy electro stompers, sometimes with thrilling guitar solos, sometimes with basslines you can feel from space.
Best bits:
- ‘Thru And Thru’ – best closing track on any album this year, bar none.
- ‘Mindkilla’ – shaking the foundations with that bassline.
- ‘Adult Goth’ – a floor-filler from a dimension where floor-fillers are made by super-smart computers not slavering morons.

5 PJ Harvey – Let England Shake

Oh come on, of course this was going to be in here. It’s ruddy, bloody marvellous, with the emphasis on blood because this is one of those albums where EVERYONE DIES! It is, frankly, the best album of the year. Why is it not at #1 in that case? Only because it’s heavy as anything, and I sometimes can’t listen to it for the same reason I am not always in the mood to watch Grave Of The Fireflies. EVERYONE DIES!!! But yeah, you’ve read a million reviews of this, it’s visceral, it’s heartfelt, it’s brilliant.
Best bits:
- ‘All And Everyone’ – the way the track builds and threatens a cathartic release but then… doesn’t. Only more misery. Brutally effective.
- ‘The Last Living English Rose’ – swaggering and proud musically, broken and crushed lyrically, a beautiful juxtaposition.
- ‘The Glorious Land’ – for those painful lyrics – “What is the glorious fruit of our land?/Its fruit is deformed children”.

6 Nicola Roberts – Cinderella’s Eyes

Yeah, that’s right, a Girls Aloud solo album. Nicola was clearly always the best member of GA, and by making an album with Metronomy, Diplo and Dragonette she’s shown that it needn’t be death by Will.I.Am and auto-tune robot vocals. What Nicola has realised is that all the best pop music is really really sad. And god, taken purely on its words this is a contender for the most misery-filled album of the year, with lots of lovely minor key synth pop and occasionaly shonky piano ballads driving it forward so you don’t end up wanting to kill yourself just by listening.
Best bits:
- ‘Gladiator’ – surprisingly lyrical dexterity and freakiness (Nic can apparently fire bullets from her chest, who knew?) with a nice martial guitar strut.
- ‘Yo Yo’ – top notch sad pop.
- ‘Beat Of My Drum’ – initially disorientating but after a few listens it’s great in a MIA-without-the-edge-or-the-bouts-of-cringey-stupidity way.

7 Young Galaxy – Shapeshifting

Young Galaxy are a bit hard to describe. They’re not really electro but they use a lot of synths to twinkle and parp. They’re kind of indie, in that they make songs of conventional structure and use guitars to jangle sometimes. They certainly like their percussion, and some songs are just a single guitar or synth line over an array of claps, whacks and other percussive sounds (castanets, steel drums). They have moments of small-scale Arcade Fire-ness. Ok, they have quite a few Arcade Fire style moments, albeit with fewer instruments going off at once. Then again they are Canadian. Must be in the water up there.
Best bits:
- ‘Peripheral Visions’ – if only for the big sing-a-long at the end which is quite affecting.
- ‘Blown Minded’ – stately march which nicely shows off the whole epic-but-kind-of-minimalist vibe of the whole album.
- ‘We Have Everything- – unashamedly anthemic.

8 Those Dancing Days – Daydreams And Nightmares

Adorable Swedish scamps who look like a bunch of off duty Brownies and sing sweet little ditties about… threatening violence against crap boyfriends and bitchy girl rivals alike. Wait, that can’t be right. Oh but it is, TDD serve up ADHD nuggets of joyful indie-pop, none of which overstay their welcome. It’s as if they need to wrap it up quickly and get home for tea and biscuits. I always find myself loving one album a year purely because it does a simple trick very very well, and this year it’s TDD. There’s nothing revolutionary here, but they have a scary mastery of their chosen form. Plus they’ve split up now, so no more fun for us. :(
Best bits:
- ‘Fuckarias’ – there’s no way this track should sound as muscular as it does, like being chased by a big pink truck.
- ‘I Know Where You Live pt.2’ – brilliantly huge chorus.
- ‘Can’t Find Entrance’ – breakneck speed sugary indie joy.

9 Washed Out – Within And Without

As the man who spearheaded the terribly named ‘chillwave’ invasion of the music blogosphere a couple if years back, there was a good chance that Ernest Greene, aka Washed Out, might have left it a year too late with this debut. The good news for Greene, and us, is it’s never too late for good albums and this is a very good album. It sounds suitably fuzzy, as if you’re remembering these songs rather than listening to them, with vocals floating around in the ether. But it’s not indulgent, and it doesn’t lack melodies, Greene grasping that lovely sonics mean little if they’re just tapering over a lack of decent tunes. There’s even a couple of sing-a-longs if you can decipher the words. Maybe it’s more fun to sing along in a fuzzy, distorted style.
Best bits:
- ‘You And I’ – simply gorgeous diet with Caroline from Chairlift, which just oozes class.
- ‘Amor Fati’ – gorgeous jangler.
- ‘Eyes Be Closed’ – keyboards splashing around like the ocean.

10 Harrys Gym – What Was Ours Cannot Be Yours

More sad songs from the melancholic Norwegians. Harrys Gym like to bring along a dash of electronics to their eleagic little epics, each building up and spiralling down, like tides. The record is centred around Anne Lise Frøkedal’s beautiful voice, all poised sadness or restrained sexiness, harmonising over herself. It’s an album which seems fixated on the inability of humans to be strong or heroic all the time. Musically there’s a lot of heft to the surroundings, tricks stolen from dance, trip hop and rock are peppered around the place, rhythms juddering or basslines crunching underneath Frøkedal’s latest missive about someone having the spirit crushed out of them by age or feckless partners or the weight of expectation. Heavy and light at the same time.
Best bits:
- ‘Sailing Home’ – specifically at 1:34 when the bass and the drums pile in properly.
- ‘Old Man’ – for one of the most heart rending vocals of the year.
- ‘No Hero’ – the slightly odd pairing of the indie guitars, the wobbling synth bass and the vocals shouldn’t work but does, very well.

December 30, 2011

Hollyzone's Albums Of 2011 11–28

For part one we have 11-28, the albums which were very good, but not quite good enough. Except for some which were good enough only someone else made something which was even gooderer. Anyway, there are 28, which isn’t a nice round number unless you have 14 fingers in which case it is. Sorry. I couldn’t be arsed to write about any of the other albums I’ve heard this year which means that they either weren’t very good, they were disappointing, or I haven’t heard enough of them (Ghostpoet, Lanterns On The Lake, I Break Horses and Soft Metals need to be filed under this category). Part two with a more numerically coherent top ten is tomorrow. Lucky you. You’re welcome.

11 St Vincent – Strange Mercy

Annie Clarke is a rock queen! No really, she is. She may not be death metal, but she’s throwing some guitar shapes here, alongside the stories about smalltown American scandals, sex, angry young women, sex, and lovers being shitty to each other. And sex. It’s a pretty lush album, each track a little wall of sound in itself, with strings, brass, wobbly electro basslines, and choirs of Clarkes all jostling for space alongside the guitars. Oh the guitars. This is more proof (alongside Wild Flag, below) that you don’t have to play every note on the damn guitar super fast to show you’ve got a mastery over it. Fun, thrilling, primal, sophisticated riffs are allowed too. Maybe it helps being a girl with a guitar, and if so, St Vincent is the head girl.
Best bits:
- ‘Surgeon’ – for that spiralling outro of layered vocals and the dizzying bassline.
- ‘Northern Lights’ – fuzzy chug-a-long rock out.
- ‘Cruel’ – starts orchestral then gets its claws out.

12 Lykke Li – Wounded Rhymes

Lykke Li has the sort of voice you’d expect to hear in an old 60s recording, and there’s a lot of almost retro sounding instruments and songs on display here. “Almost” retro because they’re not. This isn’t yet another album with a tedious obsession with olden days music (as so many of those desperate to touch the greatness of Winehouse have attempted) but one where those old style sounds are moulded into lovely, up to date numbers. It’s classy in the way that everyone pretends pop used to be (it wasn’t) hence why it doesn’t really sound like pop any more. So if you want to pretend that things were better in the old days slap this on and pretend its 1965 and we’re all so hip and happening and things are better. Just don’t look too shocked when she sings about prostitutes and other things which would have ensured this record would never have been released in 1965.
Best bits:
- ‘Youth Know No Pain’ – wurlitzer whirls, everyone gets up and dances.
- ‘I Follow Rivers’ – the percussion on this alone is worth the price of the album (unless you downloaded it for free in which case it’s worth more).
- ‘Sadness Is A Blessing’ – how was this not number 1 all year? It would have been if Adele had released it.

13 Metronomy – The English Riviera

Apparently it’s a concept album about the English seaside except it really isn’t. Instead it’s more falsetto harmonies, more quirky little ditties, and more evolution for the band who still sound like they’re recording everything in their bedroom, in a good way. There’s more heft in the rhythm section this time around which is welcome without detracting from the charm, and it’s a thoroughly well-crafted piece of sophisticated pop elegance. It nicks from 80s smooth pop but without sounding like it should be soundtracking wankers snorting coke in yachts while wearing white jeans. No, it’s geek-indie at its best.
Best bits:
- ‘She Wants’ – sinister slap bass pushing everything forward.
- ‘The Bay’ – smooth as hell pop which strays dangerously close to 80s wanker-pop without falling into those deadly waters.
- ‘Everything Goes My Way’ – all Metronomy albums must have a wonky duet crammed full of charm, this is it.

14 Cold Cave – Cherish The Light Years

You what’s wrong with most 80s influenced bands these days? Irony. Now don’t get me wrong, I love irony, but the best things about 80s music was that everyone took themselves very very seriously, especially the synth pop brigade. Cold Cave take themselves insanely seriously. It’s proper synth pop iconagraphy ahoy with lyrics about graveyards and anguished lovers either facing impossible odds together or breaking up in melodramatic waves sadness. It’s all delivered with the subtly of a North Korean military parade, waves of drums and keyboards and bellowing. There’s not a lot of light and shade, just a juggernaut of seriousness, but the sheet sincerity of it all is infectious and endearing. I can’t resist its silliness and I advise you don’t either.
Best bits:
- ‘Catacombs’ – totally seriousface, very silly to start with, but by the end it’s sucked you in so much that the payoff is quite emotional.
- ‘Underworld USA’ – very 80s, nice guitars, silly serious sing-a-long chorus.
- ‘Alchemy And You’ – best use of trumpets 2011.

15 Wild Flag – Wild Flag

Ah, y’know what, let’s not even begin to pretend this doesn’t sound loads like Sleater-Kinney. Wild Flag have Carrie and Janet from S-K, plus mates, and they have made a record which sounds an awful lot like S-K would have done had they decided to follow up The Woods with an album featuring more big, raw rock songs with twiddly (but not fretwanky) guitars, raw and zesty vocals, and retro, kind of 60s sounding keyboards. It is big, it is clever, it is endearingly life-affirming, with blasts of handclaps, 60s girl-group style backing vocals, and a pleasingly punkish edge to everything.
Best bits:
- ‘Romance’ – exactly two minutes in when the music drops out and it’s all handclaps. That.
- ‘Boom’ – for the breakneck crunching guitars.
- ‘Future Crimes’ – the contrast between the track’s general urgent heaviness and the twinkling keys.

16 EMAPast Life Martyred Saints

Droney, fuzzy, scuzzy, it’s like Past Life Martyred Saints is making a case for “dirgey” to be reclaimed as a positive rather than negative adjective. Quite a lot of the album is heavy, not in a heavy metal way, but in an oppressive and all consuming way, like a humid evening. That’s another word which is appropriate here if we can accept it as a positive not a negative – oppressive. There’s something enveloping about Erika M Anderson’s world, with its tales of small town freaks, angry geeks, and ominous metaphors. There are also plenty of squalls of guitar noise, some of it stately, some of it pure rocking out. It’s shit. Sorry, it’s THE shit. Now there’s a negative adjective turned positive for you.
Best bits:
- ‘The Grey Ship’ – an epic in two parts, a swaying acoustic beginning and a massive rockout ending.
- ‘Milkman’ – distorted glam stomper.
- ‘California’ – fuzzy rant, less a song, more a thrilling sermon with guitar noise.

17 Grimes Geidi Primes/Halfaxa

It was quite odd seeing NME calling Grimes the future of dance music, if only because it’s not that easy to dance to. At all. Yeah I tried. I’m being cheeky and lumping these two albums together which probably isn’t fair. Also she released them as free downloads in 2010, but as they only came out on record this year I am counting them. They’re worth counting too, two albums of dreamy, fuzzy electro. That’s ‘dreamy’ as in the full range of dreams, from twinkling brief half songs, through to terrifying nightmares of sinister little-girl-lost vocals and disconcerting waves of synth. Both albums are hodge-podges of ideas in the best possible way.
Best bits:
- ‘Weregild’ (from Halfaxa) – if only for the bit where the drums kick in.
- ‘Swan Song’ (from Halfaxa) – Crystal Castles if they were dreamy rather than furious.
- ‘Rosa’ (from Geidi Primes) – almost a proper song, twangy guitar and nearly intelligible lyrics, but what’s best about it that it’s groovy.

18 tUnE-yArDs – W H O K I L L

Ah, the unmistakable sound of drums and a car horn. A ragged guitar accompanied by the sound of falling wood. A raw voiced and angry woman harmonising with a police siren. Who knew Little Mix’s album would be so adventurous? Jokes, Merrill Garbus wouldn’t get past the first round of X Factor, if only because rather than some dirgey ballad her audition piece would be like one of the tracks off W H O K I L L, a righteous but often funny rant about the complete shitness of modern life, backed by a percussive cascade. Under Garbus’s command all instruments, including guitars, saxophones and, yes, sampled sirens, are merely shards of percussion to be used like some modern approximation of tribal rhythms.
Best bits:
- ‘My Country’ – a cascade of instruments and vocals, all falling down a hill.
- ‘Gangsta’ – strutting, jerky barrage of noise which somehow resolves itself into a cautionary tale with a sing-a-long chorus.
- ‘Bizniss’ – the lyrical and vocal dexterity here makes it a winner.

19 Zola Jesus – Conatus

Lots of heavy percussion, thudding towards the listener like a relentless army of goth girls, each with a cryptic story of vague sadness to impart. And yet, there’s also a lot of arms in the air rave moments, albeit slowed down from the high tempos of dance music, giving the whole album a weirdly underwater mood to it. Dark and mysterious but still inviting all the neighbours around for a party with black cake, black balloons and black party bags.
Best bits:
- ‘Vessel’ – the mechanical percussion is great.
- ‘Ixode’ – spiralling, layered vocals lead to a thoroughly ecstatic conclusion.
- ‘Seekir’ – this could almost be a pure pop song if it weren’t for the weird backing vocals, but when the drums kick in the dancing’s good.

20 Katy B – On A Mission

Ok, so she’s pop as anything, this isn’t edgy dubstep, and there’s the end of the last track where she goes off on one like an Oscar winner thanking her parents, cat, local lollipop lady for making it all happen, but hey, here’s a good dance record in a year of bad dance records. It helps that Katy has a good voice, distinctive even if the lyrics are a little banal, albeit in a sassy way. Thing is, when this album is good, it’s very very good, exhilarating to dance to, and fun to listen to. And yes, it’s not an album of great lyrical insights, but it’s nice that the overall theme is a girl being confident and honest about wanting a good time on her terms.
Best bits:
- ‘Katy On A Mission’ – huge wub wub wub anthem, resistance is futile.
- ‘Witches Brew’ – captures all the worst keyboard parts from late 90s trance and makes them good again using bleepy bleeps.
- ‘Broken Record’ – the last 45 seconds are possibly the best fade out ending in ages.

21 British Sea Power – Dancehall Valhalla

In which British Sea Power make another British Sea Power record. If that means anything to you then it’s an endorsement. If not then let me explain briefly – someone forgot to tell British Sea Power about any and all developments in indie since 2001. Therefore they make quirky guitar led songs about esoteric lyrical themes (this time round is more military themes, more star gazing, and more celebrations of music itself). These songs are then polished with big epic guitar riffs, and occasional deviations into slightly leftfield areas, but always returning to the slightly unfashionable indie underneath. And then they stick an almost 12 minute wigout at the end. Typical.
Best bits:
- ‘We Are Sound’ – top notch outro, classic BSP tactic of piling more and more sheer stuff in til the song pops.
- ‘Mongk II’ – twisted vocals over a driving insistent beat.
- ‘Observe The Skies’ – the most BSP song on here.

22 Chelsea Wolfe – Apokalypsis

Super-sinister, droning, hypnotic, it starts with a 23 second cover of a death metal song, and then soundtracks the most nightmare waltz through a haunted house you could imagine. Even the songs which don’t sound like nightmares are a bit off, love songs about being weirdos freaks. Glacial beauty, but only if the glacier is made of black ice.
Best bits:
- ‘Mer’ – guitar riffs strangely reminiscent of Brand New at their most haunting.
- ‘Tracks (Tall Bodies)’ – the most unnerving love song of the year, without being at all explicit.
- ‘Moses’ – builds around an unending creeping guitar riff into a woozy finale.

23 Bjork – Biophilia

A very sparse album which actually requires some concentration, but it’s worth it for the rewards. Of course Bjork would never do something as dull as simply release an album of songs you can sing along to, and this isn’t really a collection of songs per se, more some interesting sonic experiments with moments of startling beauty interspersed between moments of “WTF?” and “huh wuh?”. Guaranteed not to be to everyone’s tastes.
Best Bits:
- ‘Crystalline’ – surprise drum’n’bass outro.
- ‘Cosmogony’ – like a really really weird Disney song.
- ‘Mutual Core’ – I love the ‘chorus’ (insofar as there is one) on this.

24 Neon Indian – Era Extraña

Could Alan Palomo have made it more obvious than starting this album with what sounds like Space Invaders launching? I wasn’t that impressed with his debut, but this follow up is a great slice of electro-pop, layering 80s computer game noises over Palomo’s chillwave-ish vocals. It sounds like a distant Wayne Coyne trying to seduce a room full of retro-gamers, probably unsuccessfully because they’re all trying to beat Donkey Kong and save the princess.
Best bits:
- ‘Polish Girl’ – chillwave with bleeps not woozy guitars.
- ‘Fallout’ – see above.
- ‘Suns Irrupt’ – see above… look, they all sound the same but I like it.

25 Lady Gaga – Born This Way

It’s too long, there’s too much filler, one track sounds like Shania Twain, but in the end there’s a reason the Lady is a pop juggernaut crushing all before her and that’s because when she gets it right, she gets it righter than most. The sheer OTT-ness of the album sometimes works a total treat and it is nice to hear a pop star wanting to talk about more than just being ‘in da club’. We all know what it sounds like, and I am cool with that.
Best bits:
- ‘Government Hooker’ – precision engineered for the dancefloor.
- ‘Judas’ – pop as if made by evil robots who will crush and enslave us all, but in a good way.
- ‘Edge Of Glory’ – we’ll regret it in a few years time when they are everywhere, but here is a sax solo which is great!

26 True Widow – An High As The Highest Heavens And From The Centre To the Circumference Of The Earth

Slowly slowly the downtuned guitars and sludgy bass unfurl into… something. Not really songs because they’re more like funeral grooves, as if someone decided to slow down a load of danceable songs then cover them using only guitars dug from prehistoric times. Whack a load of floating and distant vocals on top and voila. An album of gloom which isn’t oppressive. An album where all the songs seem to sound the same, but they aren’t, and besides too much variation would ruin the mood. Which is one of futile despair. One for dinner parties and social gatherings.
Best bits:
- ‘Jackyl’ – for the droning guitars and spooky vocals.
- ‘NH’ – for the droning guitars and spooky vocals.
- ‘Boaz’ – for the amazing drum and bass meets opera bit in the… kidding, it’s for the droning guitars and spooky vocals.

27 Maybeshewill – I Was Here For A Moment Then I Was Gone

More heavy rock meets post-rock from Maybeshewill, the band who like to make post-rock songs but either cut out the first five minutes of build up and just slam in the bit with the loud guitars and drum fills, or they condense it all down into four minutes. It’s definitely worth it for people who like waves of uplifting guitars to wash over them, and I am one of those.
Best bits:
- ‘Red Paper Lanterns’ – for the best in guitar riff and xylophone interactions.
- ‘Critical Distance’ – the lovely cascading piano could have been Coldplay at their best but instead it gets to play with with skittering drums and big guitars and souds all the better for it.
- ‘Farewell To Sarajevo’ – stately and pretty.

28 Yacht – Shangri-La

DFA’s oddballs return with another album of strangeness which could be self-help motivational indie-dance, or an ironic act of such subtlety that it’s impossible to know who the joke is on. Whatever they’re on (and I think they’re being sincere) they’ve added some decent tunes, all in typical DFA style. It’s not up there with LCD Soundsystem, but in that vein cowbells are whacked, beats are dropped, basslines are wobbled and the pair have a giddy enthusiasm which means you’re about halfway through the album before you realise most of what they’re saying is bollox, but it’s danceable and fun bollox.
Best bits:
- ‘Dystopia’ – for sheer balls at nicking the chorus from elsewhere and the lolling rhythm.
- ‘I Walked Alone’ – for using autotune in a way which isn’t irritating.
- ‘Tripped And Fell In Love’ – the longest track, a Juan Maclean-ish groove.

December 20, 2011

UKOUG 2011

Another year of UKOUG conference closed its curtain. I attended all of the four days session, including the oaktable Sunday afternoon. I lost trace of how many times I went to this year end festival.
I now have some time to reflect what’s good :

Good Presentations:
A Deep Dive into the SQL monitoring Report by Greg Rahn
Oracle Optimizer: Upgraing to 11g without Pain by Maria Colgan
Oracle optimizer: Best Practices for Managing Optimizer Stats by Maria Colgan
Implementing Effective Database Auditing in 3 steps

Not too bad
user Experiences with Data Guard Fas-start Failover
Net Services – Best Practices for Performance, Scalability & High Availability by Kussi Mensah

Round Tables
Oracle scurity
RAC and HA roundtables

Keynote by Cary Millsap started to become interesting when I planned to leave. I ended up finished it.

The last day of the conference is not the best day. So if you can only go to conference for one day, don’t choose the last day.

November 13, 2011

New winter kit.

My new, shorter commute means I can now largely do away with my bike-specific kit, and ride in to work in “normal” clothes. Doubtless a relief for those who don’t have to suffer the sight of my lycra-centric outfits any more, but it has required some thought as to exactly what to replace then with.

So I was very pleased when the nice folk at Berghaus asked me to review some outdoor clothing which is just the ticket for a few miles riding through town on an autumn morning. First up, this:
Ardennes softshell
A splendid softshell jacket. It’s warm, windproof, waterproof-ish (the seams aren’t sealed, so I wouldn’t rely on it for a long time in really foul weather, but it’s stood up to 20 minutes or so of steady drizzle just fine). The cut is slightly more relaxed than a dedicated biking jacket (meaning it looks just fine down the pub), but has enough length in the arms and the back to avoid any annoying cold spots when leant over the handlebars. If I were being really picky, I could note that it doesn’t have pit-zips or any other kind of vents, which means it can get a bit wam if you’re working hard (e.g. when racing roadies on the way into work) – so far the weather’s still too warm to wear it for “proper” long rides. It’s also not very reflective, so you need something high-vis to slip over the top now the nights are dark.
But for riding to work it’s been ideal, and it’s also been great for standing on the sidelines of windswept football pitches watching the kids – which at this time of year is a pretty stern test of any fleece!

Item #2 on the list is a new rucksack – specifically, a Terabyte 25l .
Terabyte 25l
As the name suggests, this is optimised for carrying your laptop to work, rather than your ice axes into the northern Cairngorms or your inner tubes round Glentress. It features an impressively solid-feeling padded sleeve which will comfortably take anything from an ipad to a big-ish laptop and hold it securely, as well as a main compartment big enough for a packed lunch and my running kit, and the usual assortment of well-laid-out pockets and attachments. I particularly like the little strap on the back of the pack for attaching a extra bike light to. It’s comfortable even when loaded up, and plenty stable enough to bike with. Highly recommended.

October 28, 2011

Scala: 3 months in

So, I’ve been using scala in production for a few months now. Is it everything I expected it to be?

First, a little bit of background. The project I’m working on is a medium-sized app, with a Scala back-end and a WPF/C# fat client. The interface layer is a set of classes autogenerated from a rubyish DSL, communicating via a combination of client-initiated JSON-RPC over HTTP, and JSON-over-AMQP messages from server to client. It’s split into about half a dozen projects, each with it’s own team. (I should point out that I have little involvement in the architecture at this level. It’s a given; I just implement on top of it.)

The server side is probably a few thousand classes in all, but each team only builds 500 or so classes in its own project. It integrates with a bunch of COTS systems and other in-house legacy apps.

In functionality terms, it’s really fairly standard enterprisey stuff. Nothing is that complicated, but there’s a lot of stuff, and a lot of corner-cases and “oh yeah, we have to do it like that because they work differently in Tokyo” logic.

So, Scala. Language: Generally lovely. Lends itself to elegant solutions that are somehow satisfying to write in a way which java just isn’t. Everything else? Not so much. Here are my top-5 peeves:

  • The compiler is so slow. For a long time, the Fast(er) Scala Compiler simply wouldn’t compile our sources. It threw exceptions (no, it didn’t flag errors, it just died) on perfectly legitimate bits of code. This meant using the slow compiler, which meant that if you changed one line of code and wanted to re-run a test, you had to wait while your 8-core i7 churned for 2-3 minutes rebuilding the whole world. In a long-past life, I cut my teeth writing mainframe cobol with batch compilation. This was like that, all over again. TDD? Not so much. Recently the FSC has improved to the point where it can actually compile code, which is nice, but sometimes it gets confused and just forgets to recompile a class, which is hella confusing when you’re trying to work out why your fix doesn’t seem to be having any effect.
    I would gladly exchange a month of writing WPF automation code (that’s a big sacrifice, for those who haven’t endured it) for the ability to have eclipse’s hot-code-replace, unreliable as it is, working with scala.
  • The IDEs blow. Eclipse + ScalaIDE just plain doesn’t work – apart from choking on our maven config, even if you manually build up the project dependencies, it will randomly refuse to compile classes, highlight errors that don’t exist (highlighting errors on lines that contain no code is a personal favourite), and ignore actual errors. IDEA + Scala Plugin is better – this is now my day-to-day IDE, despite starting out as a dyed-in-the-wool Eclipse fanboy, but it’s slow and clunky, requires vast amounts of heap, and the code completion varies from arbitrary to capricious.
  • Debugging is a pain. Oh yes,
    data.head.zipWithIndex.map((t) => data.map((row: List[Int]) => row(t._2)).reverse)    

feels great when you write it, but just try stepping through that sucker to find out what’s not working. You end up splitting it up into so many temporary variables and function calls that you might as well have just written a big nested for-loop with couple of if-elses in it. Sure, you can take them all out again when you’ve finished, but are you really helping the next guy who’s going to have to maintain that line?

  • The java interop is a mixed blessing. Yes, it’s nice that you have access to all that great java code that’s out there, but the impedance mismatch is always lurking around causing trouble. Of recent note:
    – Scala private scope is not sufficiently similar to java private scope for hibernate field-level access to work reliably
    – Velocity can’t iterate scala collections, nor access scala maps.
    – Mokito can’t mock a method signature with default parameter values.

The list goes on. None of these are blockers in any sense – once you’ve learned about them, they’re all pretty easy to work around or avoid. But each one represents an hour or so’s worth of “WTF”-ing and hair-pulling, and these are just the ones that are fresh enough to remember. Scala-Java interop is a constant stream of these little papercuts.

  • Trivial parallelisation is of minimal value to me. Sure, there are loads of cases where being able to spread a workload out over multiple threads is useful. But, like a great many enterprise apps, this one is largely not CPU-bound, and where it is, it needs to operate on a bunch of persistent objects within the context of a JTA transaction. That means staying on one thread. Additionally, since we’re running on VMWare, we don’t have large numbers of cores to spread onto, so parallelisation wouldn’t buy us more than a factor of 2 or 3 in the best case. In a related vein, immutable classes are all well and good, but JPA is all about mutable entities. There have been a few bits of the design where baked-in immutability has led to a cleaner model, but they’re surprisingly few and far between.

Not that long after I started work on this project, there was a video doing the rounds, showing how Notch, the creator of Minecraft, used Eclipse to create a pretty functional Wolfenstein clone in java in a 48-hour hackathon. Whilst that kind of sustained awesomeness is out of my reach, it illustrates the degree to which the incredibly sophisticated tooling that’s available for Java can compensate for the language’s shortcomings. If you haven’t watched it, I recommend skimming though to get a sense of what it means to have a toolset that’s completely reliable, predictable, and capable of just getting out of the way and letting you create.

If I were starting this project again, I’d swap the elegance and power of the scala language for the flow of java with a good IDE. Even if it did mean having to put up with more FooManagerStrategyFactory classes

September 07, 2011

5 Operations metrics that Java web developers should care about

Sometimes it’s nice, as a developer, to ignore the world outside and focus in on “I’m going to implement what the specification says I should, and anything else is somebody else’s problem”. And sometimes that’s the right thing to do. But if you really want to make your product better, then real production data can provide valuable insights into whether you’re building the right thing, rather than just building the thing right. Operations groups gather this kind of stuff as part of normal business*, so here are a handful of ops. metrics that I’ve found particularly useful from a Java Web Application point of view. Note that I’m assuming here that you as the developer aren’t actually responsible for running the aplicationp in production – rather, you just want the data so you can make better informed decisions about design and functionality.

How should you expose this information? Well, emailing round a weekly summary is definitely the wrong thing to do. The agile concept of “information radiators” applies here: Big Visible Charts showing this kind of data in real time will allow it to seep into the subconscious of everyone who passes by.

Request & Error rates

How many requests per second does your application normally handle? How often do you get a 5-fold burst in traffic? How often do you get a 50-fold burst? How does your application perform when this happens? Knowing this kind of thing allows you to make better decisions about which parts of the application should be optimised, and which don’t need it yet.
Requests per minute graph
Request rates for one node on a medium-scale CMS. Note the variance throughout the day. The 5AM spike is someone’s crazy web crawler spidering more than it ought to.

Error rates – be they counting the number of HTTP 500 responses, or the number of stack traces in your logs, are extremely useful for spotting those edge-cases that you thought should never happen, but actually turn out to be disappointingly common.
Error rates graph
Whoa. What are all those spikes? Better go take a look in the logs…

GC Performance

GC is a very common cause of application slowdowns, but it’s also not unusual to find people blaming GC for problems which are entirely unrelated. Using GC logging, and capturing the results, will allow you to get a feel for what “normal” means in the context of your application, which can help both in identifying GC problems, and also in excluding it from the list of suspects. The most helpful metrics to track, in my experience, are the minimum heap size (which will vary, but should drop down to the same value after each full GC) and the frequency of full GCs (which should be low and constant).

weekly GC performance
A healthy-looking application. Full GCs (the big drops) are happening about one per hour at peak, and although there’s a bit of variance in the minimum heap levels, there’s no noticeable upwards trend

Request duration

Request duration is a fairly standard non-functional requirement for most web applications. What tends not to be measured and tracked quite so well is the variance in request times in production. It’s not much good having an average page-load time of 50 milliseconds if 20% of your requests are taking 10 seconds to load. Facebook credit their focus on minimising variance as a key enabler for their ability to scale to the sizes they have.
Render performance graph
the jitter on this graph gives a reasonable approximation of the variance, though it’s not quite as good as a proper statistical measure. Note that the requests have been split into various different kinds, each of which has different performance characteristics.
smokeping screenshot
Request speed with proper variance overlaid

Worker pool sizes

How many apache processes does your application normally use? How often do you get to within 10% of the apache worker limit? What about tomcat worker threads? Pooled JDBC connections? Asynchronous worker pools? All of these rate-limiting mechanisms need careful observation, or you’re likely to run into hard-to-reproduce performance problems. Simply increasing pool sizes is inefficient, will more likely than not will just move the bottleneck to somewhere else, and will leave you with less protection against request-flooding denial of service attacks (deliberate or otherwise).

Apache workers graph
Apache worker pool, split by worker state. Remember that enabling HTTP keepalive on your front-end makes a huge difference to client performance, but will require a significantly larger pool of connections (most of which will be idle for most of the time)

If you have large numbers of very long-running requests bound by the network speed to your clients (e.g. large file downloads) consider either offloading them to your web tier, using mod_x_sendfile or similar. For long-running requests that are bound by server-side performance, (giant database queries or complex computations), consider making them asynchronous, and having the client poll periodically for status.

Helpdesk call statistics

The primary role of a helpdesk is to ensure that second and third-tier support isn’t overwhelmed by the volume of calls coming in. And many helpdesks measure performance in terms of the number of calls which agents are able to close without escalation. This can sometimes create a perverse incentive, whereby you release a new, buggy version of your software, users begin calling to complain, but the helpdesk simply issue the same workaround instructions over and over again – after all, that’s what they’re paid for. If only you’d known, you could have rolled back, or rolled out a patch there and then, instead of waiting for the fortnightly call volumes meeting to highlight the issue. If you can get near-real-time statistics for the volume of calls (ideally, just those related to your service) you can look for sudden jumps, and ask yourself what might be causing them

Helpdesk call volumes

* “But but… my operations team doesn’t have graphs like this!” you say? Well, if only there were some kind of person who could build such a thing… hold on a minute, that’s you, isn’t it? Building the infrastructure to collect and manage this kind of data is pretty much a solved problem nowadays, and ensuring that it’s all in place really should be part and parcel of the overall development effort for any large-scale web application project. Of course, if operations don’t want to have access to this kind of information, then you have a different problem – which is topic for a whole ‘nother blog post some other time.

August 13, 2011

A blogging good wedding

So, yesterday Sam and Lisa got married. I would post pictures but I forgot my camera. They met through Warwick blogs, and it is also how I met them, my boyfriend and a load of other friends that were there too.
Anyway as nobody else has blogged about it on here I thought I should.

August 12, 2011

Collective Punishment And The Riots

Writing about web page http://www.independent.co.uk/news/uk/crime/councils-set-to-evict-rioters-2335878.html

Is anyone else slightly scared by the threats from several councils to evict people found guilty of rioting this week from their council homes? In a statement picked up by the Independent, Councillor Paul Andrews, Manchester City Council’s executive member for neighbourhood services, said:

“If you are a tenant of any of our properties, and you or your children are found to be involved in the looting we will use whatever powers are available to us to make sure you are thrown out.”

Does this not strike anyone else as being a bit, you know, collective punishment-y? Specifically the way Andrews goes out of his way to say that even if it’s not the adults in the house but their children, it could still lead to eviction.

As we’re living in one of the more hysterical periods of public opinion I suppose I am obligated to point out the obvious, namely that I don’t condone the riots and think anyone convicted of looting should be punished in accordance to the law (attacks on the local community deserve jail). But I feel distinctly uncomfortable with the idea that we can punish people who might have had no idea that their relatives were out causing trouble, or who might have had no power to prevent their relatives from doing so had they known.

Yes, good parenting will often lead to kids who won’t do this sort of thing, but this does not mean that everyone whose kids took part is a bad person, even when in some cases they might not be top notch parents.

A little levity from the wonderful Photoshoplooter blog.

The idea that doing this will lead to people self-examining and deciding to be better people is also something of a gamble. When I first heard about the proposal it put me in mind of the Israeli policy of last decade whereby the houses of suicide bombers from the Palestinian community were bulldozed as collective punishment. It didn’t stop the suicide bombers from coming and the Israeli army recommended it end in 2005 as “the policy had little deterrent effect and inflamed Palestinian hatred”. Or the Intolerable Acts, passed by the British in America in 1774 as a reaction to the Boston Tea Party and other acts of defiance from the colonists, which provoked mass outrage across the country and lead to the American War of Independence. I find it hard to think similar won’t happen in the cases of some of the looters. They already feel detached enough from regular society to not care that they are smashing it up, making them homeless would surely only lead to more resentment and stored up trouble?

The rioters almost certainly knew they risked jail for their actions, but few would probably have anticipated losing their homes. The powers to evict clearly exist, otherwise they would not have been mentioned as an option, but it sounds like they are being called up as an extraordinary option rather than a usual part of the due process of the criminal justice system. This means anyone subject to an eviction has been specially singled out by the authorities, and this is quite likely a recipe for future antagonism and refusal to engage from the evictees.

Maybe it’s just me, but I find idea of collective punishment is somewhat gross in itself. It doesn’t even follow the judicial convention of innocent until proven guilty as it is essentially inflicting punishment on people who are innocent, alongside the guilty. We need to punish the guilty in these riots, but we need also to avoid making a whole new set of problems, both by fostering further resentment and by escalating the viciousness of our response. If the state can’t behave in a civilised way, why should the people?

August 05, 2011

There's no such thing as bad weather…

...only the wrong clothes. Continuing the camping-kit theme, let’s talk about waterproofs. If you’re going camping in the UK, it’s going to rain sooner or later. There are a few things you can do to make this not be a problem

- a tarp, or gazebo, or event shelter, or other tent-without-a-floor-or-walls, allows you to sit around, cook, and generally be outside without getting rained on or feeling quite as penned-in as sitting inside the tent does. Watch out in high winds though.

- Wet grass will soak trainers and leather boots. Get some wellies, some crocs, or some flip-flops

- An umbrella is pretty handy

- Most importantly, get some decent waterproof clothing . For summer camping, I like a lightweight jacket rather than a big heavy winter gore-tex – drying stuff out in a tent is hard (especially if it’s damp outside), so the less fabric there is to dry, the easier it’ll be. My jacket of choice at the moment is a Montane Atomic DT, but my lovely wife has been testing a The North Face Resolve
It uses a 2-layer construction with a mesh drop liner, dries fast (and the mesh liner means it feels dry even when slightly damp), breathes well, and is slightly fitted so it doesn’t look too much like a giant sack. Packs down nice and small, and, of course, it keeps the rain out. It’s cut fairly short, so if you’re out in the rain for a long time you’ll either need waterproof trousers or a tolerance for wet legs. For 3-season use, I’d say it’s ideal.

Update: We’ve had a bit longer to test the TNF Resolve jacket, and so far (after 3 months) the signs are still good. It’s stood up to some pretty torrential rain without showing any signs of weakness, and I am informed that the colour is excellent (I wouldn’t know about that, obviously). The DWR coating is still working well, which is alway a good sign. The previously-mentioned fitted cut means that it doesn’t flap about when it’s blowy, but it’s not tight or restrictive, even when wearing a fleece underneath. The only shortcoming, which is common to a lot of lightweight jackets, is that the peak on the hood isn’t stiffened at all, so if it’s really windy, it tends to get blown onto your face a bit. This isn’t really an issue unless you’re up in the hills in really foul weather, though, and of course adding a wire to the hood would make the jacket a lot less packable, so for a 3-season jacket it’s a worthwhile trade-off.

August 04, 2011

How to back up your gmail on ubuntu

Quick-and-dirty solution:

1. install python and getmail
2. make a getmail config like this:

type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = your_user_name@gmail.com
password = your_password

type = Mboxrd
path = ~/gmail-archive/gmail-backup.mbox

# print messages about each action (verbose = 2)
# Other options:
# 0 prints only warnings and errors
# 1 prints messages about retrieving and deleting messages only
verbose = 2
message_log = ~/.getmail/gmail.log 

3. enable pop on your gmail account
4. add a cronjob like this:

13,33,53 * * * * /usr/bin/getmail -q -r /path/to/your/getmail/config

You’ll end up with a .mbox file which grows and grows over time. Mine is currently at 5GB. I have no idea whether thunderbird or evolution can open such a big file, but mutt can (use “mutt -f ~/gmail-archive/gmail-backup.mbox -R” , unless you really like waiting whilst mutt rewrites the mbox on each save) , or it’s easy enough to grep through if you just need to find the text of a specific message. If you needed to break it into chunks, you could always just use split, and accept that you’ll lose a message where the split occurs (or use split, and then patch toghether the message that gets split in half.

July 23, 2011

Offensive Times Cartoon

Writing about web page http://yfrog.com/kh6omcj

I know I am a couple of days late on this but did anyone see the Times cartoon from last week commenting on the Somalia famine situation and the phone-hacking scandal? In case you didn’t here’s a shot of it:

Via twitter.com/SaiPang

It’s amazing on how many levels this is a bad cartoon, several of which spill over into being actually offensive. The editor should have sent it back with multiple revisions requested, and ideally it should have been spiked completely.

Whenever I see political and satirical cartoons I am always bearing in mind something Martin Rowson of The Guardian said in a talk I saw five years ago. Commenting on the recent controversy surrounding the Danish cartoons of Mohammed, Rowson said he felt it was the cartoonist’s job to attack those worthy of attack, and that these were the people in power, the people more powerful than the cartoonist. In the case of the Mohammed cartoons this hadn’t happened – as Mohammed is dead it wasn’t him (a powerful religious and military figure) being attacked but followers of his religion, ordinary Muslims who would be in a position of less power than the cartoonists were. Drawing George W Bush as a chimp, as Steve Bell is wont to do, is fine because Bush is a powerful man, but ordinary people with no recourse to reply are not.

And so to the Times cartoon. Obviously it is not a direct attack on the Somali people themselves, but look at the depicted children. Each identical in appearance, and looking like caricatures at that. There are almost overtones of ‘all foreigners look the same’. It doesn’t take much to make people look different in cartoons, and it doesn’t take much to make the people you are drawing look like, oh I don’t know, actual humans.

Then there’s the comparison of the famine with the phone-hacking case. Obviously both are serious. Endemic corruption in a country’s government, press and police can’t be easily dismissed as trivia and fluff despite the efforts of some, mostly journalists associated with News International and its outlets. I am not going to argue, however, that it is more serious than the famine because, quite simply, it isn’t. On the surface this makes the cartoon’s heavy-handed and poorly drawn sentiment agreeable.

But there’s more to it than that. This week only saw the declaration of famine. Official bodies have admitted this was a last resort. As the BBC reported -

The BBC’s Africa correspondent Andrew Harding says the emotive word “famine” is used rarely and carefully by humanitarian organisations, and it is the first time since 1992 that the word has been applied to a situation in Somalia.

The situation in east Africa hasn’t suddenly crept up on us. The Today program was reporting on the imminent announcement of famine a week before it was official, and the crisis itself has been developing for years in a country with no proper government and with militias keeping international aid efforts out.

Yes, the media has been awash with the phone-hacking scandal over these last few weeks, but can it be argued that it’s not a hugely important event? It’s certainly more important and more significant than the Royal wedding or Cheryl Cole’s ongoing problems with X Factor USA/Ashley Cole/breaking a nail. A truly honest cartoon would take this into account, and a truly attentive cartoonist would have highlighted the Somalia situation earlier. As it is it just comes across as a News International employee trying to divert attention from the serious wrongdoings of their bosses.

Anyway, if you only click on one link from this post, make it this one: https://www.donate.bt.com/DEC/dec_form_eaca.html?p_form_id=DEC01 Aid agencies may not currently be able to get into the affected areas, but they need to be prepared in case they do, or if the people in these areas can escape from the combination of environmental and human hostility.

July 15, 2011

Gnome 3 / gnome–shell on Ubuntu Natty

OK, so I’m not the first to do this by a long chalk, but here’s what worked for me:

1: Install the gnome 3 PPAs (c.f. http://nirajsk.wordpress.com/category/gnome-3/) :

sudo add-apt-repository ppa:gnome3-team/gnome3
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install gnome-shell
sudo apt-get install gnome-shell-extensions-user-theme

For some reason, the default theme (Adwaita) was missing, as was the default font (Cantarell). (Could be because I didn’t install gnome-themes and gnome-themes-extra – see here). You can download Adwaita from here Get Cantarell from here, copy to /usr/share/fonts/ttf, and sudo fc-cache -rv to update.
I wasn’t too keen on the fat titlebars in adwaita, so I used this to shrink them down:

sed -i "/title_vertical_pad/s/value=\"[0-9]\{1,2\}\"/value=\"0\"/g" /usr/share/themes/Adwaita/metacity-1/metacity-theme-3.xml

You can set the theme and the font using gnome-tweak-tool (aptitude install it if it didn’t arrive with gnome-shell). I’m still looking for a nice icon set; the ubuntu unity ones are a bit orange, and the gnome ones are an unispiring shade of khaki. For now I’ve settled on the KDE-inspired Oxygen (aptitude install oxygen-icon-theme) which is OK, but still doesn’t quite look right.

There’s an adwaita theme for chrome which is nice, and makes everything a bit more consistent.

The London Smoke gnome-shell theme is really nice

Switching from pidgin to empathy gets you nice, clickable message notifications, although I’d rather they were at the top of the screen than the bottom.

Aero-style snap-to-window-edge works fine, except that for nvidia graphics cards with TwinView, you can’t snap to the edges that are between two monitors. Right-click context menus look weird in a way that I can’t quite put my finger on, but they function as expected.

Other than that, it pretty much just works. The only glitch I’ve yet to work out is that spawning a new gnome-terminal window freezes the whole UI for a second or so. Not got to the bottom of why that might be yet; if I find it I’ll post something. In gnome 2 there was a similar problem with the nvidia drivers, which could be “solved” by disabling compiz, but that’s not an option here. Update it seems to be connected to using a semi-transparent background for the terminal; if I use a solid background the problem goes away.

Things I like better than Unity, after a week of playing:
– Clickable notifications from empathy. The auto-hiding notification bar at the bottom of the screen is great, although I’ve found it can sometimes be a bit reluctant to come out of hiding when you mouse over it.
– alt-tab that shows you the window title (good when you’ve got 20 terminals open). I like the ability to tab through multiple instances of the same app in order, too. To get alt-shift-tab to work, I used these instructions
– menus on the application. Fitt’s law can go hang; I like my menus to be connected to the window they control

July 06, 2011

Camping time!

It’s summer time, and that means it’s camping season again. Camping is ace, and it’s doubly ace if you have kids. Almost without exception, kids love weaselling about in the countryside, so once you’ve got them onto the campsite, they’ll amuse themselves and you can get on with the serious business of drinking beer, talking rubbish and playing with fires. What could be finer?

There’s a curious kind of Moore’s Law at the moment, as far as tents are concerned.
Every year, technology trickles down from the top-end, so low and mid-range tents get better and better. My first tent, long long ago, was pretty much bottom-of-the-range and cost about £50 (which was a lot of money for a fourteen year old in ninteen-eighty-something). It weighed approximately a tonne, leaked like a sieve, and stood me in good stead for a few years worth of adventures. Now, for half of that price you can get one of these pop up tents pop-up tent

You don’t so much “pitch” it as just take it out of the bag and stand back. It sleeps two close friends, or one with too much gear, it’s pretty waterproof, and if you peg out the guy-ropes it’s surprisingly sturdy in the wind.

Downsides? It doesn’t pack down small, and I’m not sure it would be my first choice for a week in the Lake District in November – in cold, wet conditions the single-skin design means you’ll end up damp from condensation on the inside of the tent – but for summer weekend trips it’s brilliant; fun, easy, and it costs roughly 1/25th as much as an Ultra Quasar (though if you do have £600 to spare, I can highly recommend one of those as an alternative!).

ProTip: If you want to extend the range of weather you can use this in, get a tarp and pitch it over the top of the tent, overhanging the front by a meter or two. You get an extra layer of waterproofing, and a porch so you don’t have to bring your soggy boots into the tent.

one–liner of the day: quick–and–dirty network summary

If you’ve got a solaris box with a load of zones on, you might sometimes have an issue whereby you can see that the box is pushing a lot of traffic over the network, but you’re unsure which zone(s) are responsible. Here’s a super-hacky way to get an overview:

 snoop -c 1000 | awk '{print $1}' | sort | uniq -c | sort -n

basically, catch the first 1000 packets, find the source for each one (assuming most of your traffic is outbound; if it’s inbound then print $3), and then count the distinct hosts (i.e. your zone names) and list them.

If you have a slow nameserver you may want to add “-r” to the snoop command and then map IPs to names afterwards.

June 26, 2011

Scala: fun with Lists

So, I’m slowly working my way though some of the programming exercises at Project Euler as a means of learning scala (and hopefully a little bit of FP at the same time).
Problem #11 is my most recent success, and it’s highlighted a number of interesting points. In brief, the problem provides you with a 20×20 square of 2-digit integers, and asks for the largest product of any 4 adjacent numbers in any direction (up, down, left, right, diagonally). Kind of like a word-search, but with extra maths.

For no better reason than the afore-mentioned desire to get a bit more acquainted with a functional style of programming, I chose to implement it with no mutable state. I’m not sure if that was really sensible or not. It led to a solution that’s almost certainly less efficient, but potentially simpler than some of the other solutions I’ve seen.

So, step one: Give that we’re going to represent the grid as a List[List[Int]], let’s see if we can get the set of 4-element runs going horizontally left-to-right:

    def toQuadList()= {
      def lrQuadsFromLine(line: List[Int], accumulator: List[List[Int]]): List[List[Int]] = {
        if (line.length < 4)
          lrQuadsFromLine(line.tail, accumulator :+ line.slice(0, 4))
      data.flatMap(line => lrQuadsFromLine(line, List.empty))

(“data” is the List[List[Int]] representing the grid). Here we define a recursive function that takes a List[Int], obtains the first 4-number run (line.slice), adds it to an accumulator, and then pops the first item from the list and calls itself with the remainder.
Then we just call flatMap() on the original List-of-Lists to obtain all such quadruples. We could call Map, but that would give us a list of lists of quadruples – flatMap mushes it down into one list.
It would be nice to find a way to make the if/else go away, but other than just transliterating it into a match, I can’t see a way to do it.

Given that list, finding the largest product is trivial.

   def findMaxProduct = data.map((x: List[Int]) => x.reduce(_ * _)).max

- take each Quadruple in turn and transform it into a single number by multiplying each element with the next. Then call .max to find the largest.

So now we can do one of the 6 directions. However, a bit of thought at this point can save us some time: the Right-to-Left result is guaranteed to be the same as the left-to-right, since multiplication doesn’t care about order (the set of quadruples will be the same in each direction). Similarly, the downwards result will be the same as the upwards one.

Next time-saver: Calculating the downwards result is the same as rotating the grid by 90 degrees and then calculating the left-to-right result. So we just need a rotate function, and we get the vertical results for free:

   def rotate90Deg() = {
      data.head.zipWithIndex.map((t) => data.map((row: List[Int]) => row(t._2)).reverse)

Doing this without mutable state took me some pondering, but the solution is reasonably concise. Doing zipWithIndex on an arbitrary row of the grid (I used .head because it’s easy to access) gives us an iterator with an index number in, so we can now build up a List containing the _n_th element from each of the original lists. (The outer map() iterates over the columns in the grid, the inner one over the rows in the grid))

So now we have the horizontal and vertical totals, we need to do the diagonals. It would be nice if we could once again re-use the left-to-right quadruple-finding code, so we need to get the grid into a form where going left to right gets the sequences that would previously have been diagonals. We can do this by offsetting subsequent rows by one each time, the rotating; like this:




You can see that the horizontal sequences are now the original diagonals. Initially, I started using Option[Int] for the items in the grid, so I could use None for the “question marks”. However, after more time than it ought to have taken me, I realised that using zero for those would work perfectly, as any zero in a quadruple will immediately set that quad’s product to zero, thus excluding it from our calculation (which is what we want).
The function to do the offseting is still rather complex, but not too bad (it owes a lot to the recursive toQuadList function above:

    def diagonalize() = {
      def shiftOneRow(rowsRemaining: List[List[Int]], lPad: List[Int], rPad: List[Int], rowsDone: List[List[Int]]): List[List[Int]] = {
        rowsRemaining match {
          case Nil => rowsDone
          case _ => {
            val newRow: List[Int] = lPad ::: rowsRemaining.head ::: rPad
              0 :: rPad,
              rowsDone ::: List(newRow))
      shiftOneRow(data, List.fill(data.size)(0), List.empty, List.empty)

We define a recursive function that takes a list of rows yet to be padded, a list of zeros to pad the left-hand-side with, another for the right-hand-side, and an accumulator of rows already padded. As we loop through, we remove from the remaining rows, add to the done rows, remove from the left-padding, and add to the right padding. It kind of feels like there ought to be a neater way to do this, but I’m not quite sure what yet.

To get the “other” diagonals, just flip the grid left-to-right before diagonalizing it.

Once that’s done, all that remains is to glue it together. Because I’ve been thinking in OO for so long, I chose to wrap this behaviour in a couple of classes; one for the list-of-quadruples, and one for the grid itself. Then I can finish the job with:

      List(new Grid(data).toQuadList.findMaxProduct,
      new Grid(data).rotate90Deg.toQuadList.findMaxProduct,
      new Grid(data).diagonalize.rotate90Deg.toQuadList.findMaxProduct,
      new Grid(data).flipLR.diagonalize.rotate90Deg.toQuadList.findMaxProduct).max)

June 20, 2011

The one where I fail to go on any exciting expeditions

Writing about web page http://www.gooutdoors.co.uk/thermarest-prolite-small-p147594

So, in the spirit of previous product reviews, this should have been an entry that described various recent feats of derring-do, and casually slipped in a plug for the latest bit of camping equipment that I’ve been sent to review. Unfortunately, the last few weeks have been characterised by some especially foul weather. Last week’s planned camping trip to do the Torq rough ride was abandoned, in favour of an early morning drive down on sunday, followed by three-and-a-half hours of squelching round the “short” route in driving rain.
Then tonight’s Second Annual Summer-Solstice-Mountain-Bike-Bivi-Trip was abandoned when it became apparent that the chances of a scenic sunrise were pretty much zero, whereas the chance of a night in a plastic bag in the rain on a hill were looking close to 100%.

All of which means that my shiny new toy has so far not seen much action outside of the back garden. But it seems churlish not to write something about it; so here goes. It’s a sleeping mat specifically a Thermarest Pro-Lite. Mats might seem like a pretty mundane item, but if you’re trying to travel light, either running or on a bike, then they’re a tricky thing to get right. Too minimalist and you don’t get any sleep, then you feel like crap the next morning. Too heavy, and they start to make a serious contribution to your pack weight, which matters a lot if you’re having to run up a hill or ride down one.
For a long time, my preferred option was a cut-down foam karrimat, which was a reasonable compromise, but suffered from being a bit on the bulky side, and also not terribly warm. I have an old thermarest as well, which is fabulously comfy – great for winter camping, but far too bulky for fast & light summer trips.

There will be some pics here, when it stops raining for long enough… for now, here’s a stock photo…

So, the pro-lite: Point 1; I got the small one; it’s very small. (note the artful perspective on the photo!) If you want something that will reach alll the way down to your toes (or even your knees) this isn’t it; buy the large size. I don’t mind this, though; in the summertime I’m happy with something that just reaches from head-to-thigh. Equally, it’s quite slim. My shoulders are fairly scrawny, and this just-about reaches from one to the other. If you’ve been spending longer in the gym (or the cake shop) than on the track or the turbo, then you might want a bigger size.

Point 2: It is just unbelievably compact. Really. It rolls up into something about the size of a 750ml drink bottle. Foam karimats can’t come near to this. This makes more of a difference than you might think, because it means you can get away with a pack that’s 5L or so smaller (and therefore lighter), and still fit the mat inside (keeping your mat inside your pack is a winning plan, because you can keep it dry). It’s also great if you’re backpacking on a bike, because the smaller your pack, the less it affects the bike’s handling.

Point 3: It’s as light as a foam mat. Unlike my old thermarest, there’s not a lot inside this mat, so it’s super-lightweight. Mine weighed in at a smidge over 300g according to the kitchen scales.

Point 4: Back-garden tests strongly suggest that it’s a lot more comfy than my old foam mat. I’ll report back once it’s stopped raining long enough to try it out for real!

Update #1 : Still not had the chance to take this backpacking, but a car-camping trip confirms that it’s very comfortable indeed – just as good as my old thermarest, though in a big tent you have to be careful not to roll off it in the night!