The Lotus Fax Of Knowledge

Back Then Everyone Used Lotus

Our information-led world is a beautiful thing. Facts, help, knowledge at the click of a mouse or the tap on a screen. Such a state of affairs is especially true if you work in any technology-related field where the solution to any problem you may encounter is just a carefully crafted search away, and because someone else has almost certainly faced and solved the same problem.

But how did we manage this before the age of the internet? Before everyone was online on a near-permanent connection, how did your average computer operator sat at his desk in the mid-1990s go about solving problems with no internet to speak of available to him?

I remember it well. So here is my tale of how it all happened.

Wavy Lines To Indicate Flashback Sequence

Picture the scene. I’m in my first job out of university, being paid a vanishingly low sum of money (hey, it was better than nothing) to be the office “IT Assistant” for the local branch of a large firm of insolvency practitioners. For the first time in a year my immediate boss has gone away on a two week holiday, meaning this is also the first time in a year I’ve not been subject to random data entry or spreadsheet processing jobs landing on my desk within ten minutes of the working day started. In short, for the first time in forever once I’ve completed a handful of daily admin tasks, I’m genuinely at a total loose end.

So I seize my chance to pull out a small project which has been sat on the to-do pile for some time. Automate the production of the diary.

When dealing with corporate or personal insolvencies, there are a myriad of statutory filings you have to make, reports you must prepare for a specific schedule. And historically our office had been rubbish at keeping up to date with them. Details of what was due, and when, were all kept in the bowels of a Clipper-based application called IPS of which only a handful of us in the admin team had access. So once a week the program was commanded to spit out a text file of all the statutory returns required on a case by case basis. This file was then wrestled into shape in Lotus 1-2-3 before being formatted in a manner readable by the office’s internal videotext system. And so a variety of pretty graphs could be issued to the partners to show just how many late events we were enduring.

Production of this report was not a straightforward process. The work instruction for this in the office’s ISO9001 quality manual ran to about 12 pages, step by step instructions on how to format, sort, weed out and ultimately re-export the data. It took the best part of an hour for even the most dextrous of spreadsheet wrestlers to do it. And it was ripe for automating with a macro.

Happily, this was well within my talents, as in the months I’d been there I’d become quite the wizard in the Lotus 1-2-3 macro language (everyone used Lotus in offices back then. Excel was only beginning its march to dominance in the spreadsheet market). I would joke that eventually, I’d get it to stand up and fart the national anthem on command. So with the coast clear, a whole two weeks of just being left alone to get on with stuff ahead of me, I set to work on what was set to be my greatest gift ever to the admin team.

Circular Reference in Cell B55

Only within five minutes I’d hit a snag. The text file output by the IPS application would always contain random line feed characters at random intervals. Sadly this was enough to trigger a strange assumption made by the writers of the 1-2-3 application. If you imported a text file directly into Lotus 1-2-3 containing these characters, it would assume you wanted a new page in your worksheet and create a new one on your behalf. It was a behaviour hard-coded into the program. The only way around it (we assumed) was to open the file in a text editor and manually remove all the line feeds you could find. And then rinse and repeat if it turned out you had missed any.

This, however, was not part of my plan to automate the entire process. Convinced there had to be another way to work around this, my first mission of the project was to work out what it was. Temporarily mystified, I began browsing the application help files in search of inspiration. It was here that I turned the screen which would transform my life for the next two weeks. Details of a faxback service of solutions to common technical issues. Hey, even if this didn’t contain the answer I was looking for it would be fascinating to see how it worked anyway.

So I phoned the American number given. An automated voice instructed me to enter my fax number, plus the extension number of my desk, followed by the code for the document I wanted. Or just “1” for the index. Transaction completed, I walked to the post room and waited for the phone to ring.

What emerged behind the “for the attention of the person at extension XXXX” cover sheet was what must have been a 15-page document. A full list of technical articles containing how-to guides and solution to everyday problems. It was my first ever encounter with a technical support knowledgebase. And I had a fantastic way to access it.

It took five minutes to track it down, but yes, the solution was in there. A specific document relating to my problem of line feed characters in text imports creating multiple sheets. The bump in the road had suddenly become smooth again.

Slash Command Not Recognised

What was the workaround? There may be people who had spotted it already. Open the file up, let it populate as many sheets as it required. Then select all the data, re-export and re-import. Messy, but the result was a flat file contained in a single layer that I could then set my macro to work extracting and formatting.

By the time the boss returned from holiday, waiting on his desk was bound documentation detailing how I’d implemented my one-click solution to one of the office’s messiest tasks. Not that there weren’t other bumps in the road, including one that led to me staring at the screen for three hours without typing. I had no idea how to get a spreadsheet macro to make a human decision based on the type of event it was processing (solution: do a text search on a lookup table containing the choices), but I steadily figured it out. Often with the help of my new-found bible. The Lotus Corporation Faxback Service.

Six months later I was given a login for the then-experimental Lotus Notes system the company was planning to roll out in future. The knowledgebase came bundled with the install, I discovered. I couldn’t help but think that had taken away part of the fun.

 

Fade Into Penguins Avicii

Plagiarised Penguins

The death of any performer is always a sad occasion for their fans, but no more so than when the death is that of someone still in the prime of life. The passing of Tim “Avicii” Bergling last week at the age of just 28 has prompted a re-evaluation of the catalogue of music he produced during the course of his ten years as an EDM producer. Some of his greatest career highlights will quite rightly re-appear in the charts at the end of this week. Only when considering them back to back do you appreciate just how many of the greatest pop records of the decade bore his name.

Only there is one single which was never supposed to bear his name, and indeed if he and his manager and label had had their way would never have appeared at all.

A Tale Of Two Hits

Tyro Avicii was, by his own admission, a producer and not a songwriter. He’d been signed by dance label Ministry Of Sound on the back of the instrumental works he’d been producing for his own amusement since 2006. Everyone who heard them knew they had hit potential, but at the time his library of tracks were nothing more than instrumentals with apparently random coded titles. What was needed to turn them into smash hits was a final bit of magic. To use them as the base of proper pop songs.

One such track was a catchy ditty that Avicii had entitled Penguin, simply because it was based on a simple piano riff sampled from an old Penguin Cafe Orchestra track Perpetuum Mobile. To find a suitable song to sit on top of it the label initiated a beauty contest, inviting publishers and songwriters to pitch demos in the hope of seeing their compositions recorded by one of the most talked-of upcoming dance producers. The idea was that the untouched instrumental could be circulated to select club DJs, the track gain a following, and then a commercial smash hit based on the track that everyone loved would be unveiled.

The song selected from this process was Fade Into Darkness, written by fellow Swedes John Martin and Michel Zitron, a combination which could easily have meant the song could have become a Swedish House Mafia track rather than an Avicii one. With a lead vocal recorded by up and coming Swedish singer Andreas Moe, promos were sent out, the reception was positive and a worldwide release was slated for July 2011.

Only then there was a spanner in the works. Some of the other demos of the Penguin vocal interpretations had by now begun to circulate behind the scenes. A tape of one had made its way to the offices of Syco Records, Simon Cowell’s outfit by that time searching for material which could go on a new Leona Lewis album. Collide had a strong pedigree of its own, the lyrics and melody written by Autumn Rowe and Sandy Wilhelm who had already contributed to hit singles by the likes of Rihanna and Katy Perry.

The new track (Leona’s first brand new single in almost two years) was released to radio in mid-July. At this point, a mighty row blew up.

Twitterstorm

The problem was that Collide sounded almost identical not only to Penguin but also Fade Into Darkness, its production an almost straight retread of the two Avicii tracks. Which led to questions online as to whether the Swedish producer was involved in the Leona single as well. In fact, he wasn’t, and he was none too happy. Especially as the Leona track arrived in the same week as the release of his own record, almost totally overshadowing it.

His manager and collaborator Ash Pournouri was more explicit, responding to press queries about the connection between the two records by stating:  “We were under the impression that they were going to sample the original. They ended up copying our version.”

Yes, this was a none more 2010s kind of row. One which appeared to be playing out in public via barbed tweets from the artists in question. A few days later, stung by the intense criticism which had been flying her way Leona herself weighed in with what she probably hoped was a diplomatic explanation:

Avicii’s immediate reaction? To call it out as bullshit:

This was rapidly turning into a PR nightmare, at least for Syco. Opinion online was very much on Avicii’s side with the popular view that he had been screwed over by an organisation which felt they could act with impunity. Matters were not helped by a Syco spokesperson briefing “Avicii is already credited as a songwriter on Leona’s song. It’s a case of sour grapes from Ministry Of Sound”. In response, Popbitch that week noted: “Ministry of Sound has been astonishingly successful at building up an instrumental track through the clubs and radio before releasing it with a vocal to a high chart position and, if that is what the bloke who actually penned Penguin wanted to do, that’s what should be happening.”

Indeed such was the impasse that both Avicii and Ministry Of Sound felt they had no option but to petition the courts to stop the release, The Guardian reporting a week later:

Guardian Extract

Who Breaks The Wings

Had the case proceeded it would have produced some fascinating legal arguments. Just what was it that Avicii and his team were claiming ownership of? It wasn’t the Penguin Cafe Orchestra sample, as both parties had acknowledged. That was under the control of the estate of Simon Jeffes and who were more than happy to licence the piano riff to whoever. Was it the precise use of the sample in the context of the dance track? Again, a grey area.

It had already been established that an arrangement of a piece of music was not subject to copyright. Back in the 1970s, Jonathan King sat and watched Blue Swede have a massive US Number One hit with their version of Hooked On A Feeling, a recording which used the same “ooga chaka” vocal intro as heard on his own UK hit version a year earlier. But because that was deemed an arrangement of the BJ Thomas original he was not entitled to claim ownership of it.

No, the Collide legal row appeared to be based on the intellectual property. The producers of the Leona Lewis track were accused of copying an idea, and perhaps more to the point in a way that was damaging to the interests of the originators of the concept. By this time the release of Fade Into Darkness had come and gone. Despite Avicii fans making noise about the track online, radio programmers were only interested in the Leona Lewis track and Avicii’s best prospect to date at landing a breakthrough commercial smash hit had fallen by the wayside.

Sadly before we were able to find out what the legal argument was going to be, it was all over.

Daily Mail web clipping

The result of the settlement was that Collide could be released as planned. Significantly, however, Avicii was credited not only as a composer but also as the producer (even though he’d had nothing to do with the physical creation of the Leona track). Not only that but he also received a co-artist credit, Collide eventually listed online and in the charts as “Leona Lewis & Avicii”. The single shot to Number 4 upon release, Leona Lewis’ biggest hit single in three years and notably the first ever Top 40 entry for her initially unwilling collaborator.

Oh, Sometimes I Get This Feeling

In one of those odd coincidences, Avicii’s next single was indeed a global smash hit. But it also did so in two different versions. As if to prove that a vocal track wasn’t necessarily a pre-requisite for turning an EDM club smash into a chart hit. Levels flew up the charts worldwide in its original form, its only lyrics coming thanks to the Etta James sample on which it was based. After going through the proper channels, Levels also ended up forming the basis of Flo Rida’s Good Feeling and which swiftly followed the original into the charts, the rap hit creeping to Number One in Britain in early 2012.

It would be another year before Avicii finally hit paydirt with his own material, but in early 2013 he topped the UK charts for the first time in conjunction with Nicky Romero on I Could Be The One – ironically another track which began life as an instrumental demo NickTim before having a vocal grafted on. Tellingly by the summer, he was on top of the charts worldwide with Wake Me Up, a track which had been worked on as a pop song right from the start. Avicii was still not a songwriter, but by now as a star he had people queuing up to work with him. And nobody ever ripped him off again.

Numbers Up For Sales

Here Comes The Science Bit

If mathematics sends your mind into a tailspin, then look away now. Because in this article I’m about to peel back the curtain on the precise numbers behind the sagging rump of the paid-for musical market. Ever wondered (as dedicated online fandoms are increasingly keen on doing) exactly how many copies the latest work by your idol has been selling over the past day or so? Unless you had friends in the music industry with access to detailed sales flashes, it was impossible to know for sure.

Only now it is. And it is all thanks to flagging sales numbers which are now exposed in a manner which surely nobody could ever have anticipated.

A Brief History Of Time

Like so many apps focused on the display and ordering of live and constantly changing data sources, iTunes is at heart an XML reader. Anyone who has ever examined the contents of their computer’s Music folder will understand this. The metadata for your entire musical database is contained in a single XML index file. This, incidentally, is the reason iTunes struggles with music collections more substantial than a couple of thousand tracks. That’s one huge file it has to scan through.

Even browsing the store part of the app works in the same manner. The client requests the live data from Apple; all delivered as an XML stream. This data was never a secret. Straightforward packet sniffing exposed just where the application was polling for its data. However the process of understanding this was made far simpler in 2012 when Apple made all the relevant URLs public and essentially opened the door for third-party developers to construct interfaces to the iTunes store or incorporate it into their personal projects.

The most interesting datasets are the ones iTunes uses to construct its internal charts. The lists of the biggest selling tracks or applications or movies of any given moment. During the height of the digital download era, the live iTunes charts were an invaluable guide to taking out much of the guesswork as to how the singles chart would look at the end of the week. Although the exact methodology used in its compilation has still never officially been confirmed, it has long been apparent that your position in the iTunes sales charts represented how many copies you had sold in the preceding 24 hours relative to everyone else.

The datasets never contain the exact sales figures, merely numbers that can be used to construct what iTunes calls the Popularity Bars, those curious blocks of lines which show you at a glance which tracks from an album or which episodes of a podcast are the most popular at that particular moment in time. The numbers are always expressed as a percentage of the most popular item in the category. Thus the Number One track of the moment always has a popularity index of 1.0, and if the second biggest was selling 92% of that total, it had an index value of 0.92.

A Valued Bookmark

Dedicated chart watchers have long made use of these numbers, thanks mainly to the work of the Dutchman known enigmatically as Kworb. His kworb.net website has for years been diligently scraping the sales data for every available country and presenting them as close as possible in an hour by hour analysis of what is selling and what is trending. But deep down it was still only possible to track the “popularity bars” and look at the underlying trends they represented. There was no easy way of determining the actual numbers behind the data. This was a neat bit of obfuscation which seemed to satisfy everyone.

However since the rise of the streaming services and the corresponding collapse of the download market this obfuscation no longer works as planned. It is now possible to calculate precisely how many copies a single track has sold in the past 24 hours. All thanks to the minimal levels of sales taking place at the lower end of the market.

Above is a screencap from Kworb, showing the state of the iTunes market on Saturday (10th March). I’ve chosen Saturday simply because this represents what is usually a high point in the market and gives us some meaningful data and above all fully representative data for us to use. I first performed this exercise using data from a Wednesday, figures which were so low they were barely believable.

Notice that the tracks at the base of the Top 100 are covered by just a few tiers of percentage figures, with several tracks tied on the same numbers. Indicating they have sold exactly the same number of copies as each other. The gap between each of these tiers is tier is also uniform – in this example roughly 0.0003 percentage points each time (subject to rounding errors given that the numbers only run to four decimal places).

The implication is clear. Every time those numbers change by such a small amount, this represents a difference in sales of a single copy. Armed with that knowledge it is a simple bit of maths to work out just how many copies each track has sold based on its stated percentage points:

x / 0.0003 (where x is the number given for each track’s popularity bars)

So, say we want to calculate the sales achieved in the last 24 hours by the Number 94 single Top Off by DJ Khaled we simply have to calculate:

200 / 3 = 66.666

We round up, of course, you cannot sell fractions of a copy after all. But yes, you read that correctly. The singles at the very bottom of the iTunes Top 100 on Saturday were selling just 67 copies every 24 hours. Using this same formula we can work our way up the table to get some more meaningful numbers. For example, how many copies per day was the Number 40 single selling?

861 / 3 = 287

Just 287 copies in a day to make the Top 40 of the live iTunes chart. So what about the Top 10? Last summer when asked I would suggest to people that about 5,000 copies a day were sufficient to make the iTunes Top 10. Is that still the case right at this moment?

3702 / 3 = 1234

That’s quite jaw-dropping. The Number 10 single on iTunes as of Saturday had sold only a shade over 1,000 copies in the preceding 24 hours. Has your online fandom propelled your latest release straight into the iTunes Top 10? Then, in all honesty, there still aren’t very many of them.

If we take this to the very top of the market, the Number One single – well that is even easier to work out. We already know that its sales are expressed as 1 – or 10000 using the multipliers I’ve been using for the sake of illustration. So, therefore:

10000 / 3 = 3333

3,000 copies a day, assuming this was consistent across the week, would amount to a total weekly sale from iTunes of the Number One single of around 21,000 copies. This number is entirely in line with reported figures for the top end of the sales market, proving that our maths here is sound.

5% Of Not Very Much

At this point, you start to see how problematic it is becoming to continually look to purchased sales of music as an adequate barometer of the popularity of the pop record. This simply isn’t the case anymore. It is merely the barometer of its popularity amongst the diminishing number of people who consume music in this manner. A month ago when Kylie Minogue released her exciting new single Dancing, there were a fair number of longtime fans figuratively losing their shit online when it failed to reach the published Top 40 chart. Noting that on pure sales terms the single was the 15th biggest seller of the week, this we were told was proof of the way the integration of streaming data has “ruined” the charts.

As we now know, becoming the Number 15 best seller of the week means very little. During her first week on sale, Kylie sold just 8,000 singles. Or to put it another way, only a shade over 1,000 copies a day spread across the week. That isn’t representative of anything approaching widespread popularity.

At the start of this year, Apple was busy angrily denying a report that it was planning the imminent sunset of the iTunes store, this despite the piece alleging this being quite specific as to the timescale it was to happen and just how the customer base was going to be transferred. Denials aside, there is without question no future in the market for purchased digital tracks, these unlikely to retain the kind of niche affection that physically pressed records and CDs still maintain. Even the numbers I’ve quoted above represent an ongoing and quite calamitous collapse in the size of the paid-for digital music market.

The iTunes track download no longer has the status of a mass-market product which it once enjoyed. Everyone has moved on, and we now have the detailed numbers to prove it.

Grateful acknowledgements to the poster ‘mjdangerous’ from the Haven music forums who was the first to indicate that these calculations were possible and whose work inspired this article.