The R user group at ICL

At ICL, we recently started an informal R user group. We meet up every Thursday and discuss about old and new R packages, problems to solve, tricks, good and bad coding habits and much more.

Last Thursday we had an unusual session in which we did not only talk about R but also web services and hardware devices that could be useful:

  1. plot.ly
  2. arduino vs raspberry py
  3. creating new environments and the use of the attach() function
  4. calling functions with the same name from different packages using the namespace

PLOT.LY

Plot.ly is a web service that allows you to host your plots online and share them with friends/coworkers/students/etc.

ARDUINO vs RASPBERRY PY

Arduino is micro-controller, an open-source electronics platform based on easy-to-use hardware and software. It’s intended for anyone making interactive projects.

The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse.

CREATING NEW ENVIRONMENTS AND THE USE OF ATTACH()

When a data.frame is loaded in the current environment, its columns can be loaded as variables using the function attach().

a <- data.frame( "b"=c(1,2,3), "c"=c(4,5,6), "d"=c(7,8,9) )
# by default b, c and d are not visible
b

Error: object 'b' not found
# if we attach(a)...  b, c and d become visible
attach(a)
b

[1] 1 2 3

The same can also be done by using the new.env() command.

CALLING FUNCTIONS VIA NAMESPACE

Have you ever used two functions with the same name, belonging to two different packages?

Here is an example:

library(tgp)
library(arules)

Both of these packages have an object called “lhs”.

When you load arules after tgp, the lhs function in tgp is masked by the lhs in arules. How can you keep using both function without reloading the libraries?

The answer is: use the name space!

Every package has a set of visible functions that can be called without ambiguity by using the following strategy:

name_of_package::name_of_function

For instance, in any part of your code you can call the lhs in tgp by writing

tgp::lhs
Advertisements
Posted in User Groups | Tagged | Leave a comment

MSc Hydrology study tour: Into the wildness of Cumbria

While I am rewinding mental images of the trip in my mind, my thoughts keep coming back to the film “Into the wild”. Don’t get me wrong; this was by no means that kind of trip. But as we were leaving the mass of urban concrete and approaching the north of England and famous lakes of Cumbria, I remember I was thinking that there is so much peacefulness in green fields and snow-covered hills embracing us from both sides of the road; and how much I needed all that to reset my mind after months of exhausting long working hours. Turning around to look at 19 young faces on the coach, I was only hoping that they might feel the same. This is a story of four days on the road with my PhD student Jimmy O’Keeffe, our driver Maurice and MSc Hydrology students on our study tour to Lake District.

This is not a story about all the things we saw. Though we did see loads of interesting stuff. We wandered around Oxford with David Macdonald who told us a story on the mechanism for groundwater flooding in urban areas. We visited Eden Demonstration Test Catchment where Nick Barber and Will Cleasby showed us the water quality monitoring station at the catchment outlet where they measure all the relevant parameters to monitor the impact of diffuse pollution on river water quality. We met Tony who was kind enough to show us around his farm, and who is working closely with the people on the project to implement some of the measures for pollution mitigation, such as small-scale dams and retention ponds.

Eden Demonstration Test Catchment (photo by A.Mijic)

Eden Demonstration Test Catchment (photo by A.Mijic)

We visited Cockermouth to look at the flood defense scheme Environment Agency (EA) implemented after devastating floods in 2009. We’ve seen the fancy glass panels, floodgates and world’s unique automatic self-rising flood barriers. EA staff was too busy to show us around, but as we were passing by one of the private houses, the owner kindly invited us in and explained us all the measures that were undertaken at his property, finishing his small session with showing us the album with photos from the 2009 flood, when water came up to 1.5m into his back garden.

Cockermouth Flood Defence (photo by J.O'Keeffe)

Cockermouth Flood Defence (photo by J.O’Keeffe)

We spent a lovely morning at The Blencathra Centre for environmental education with Tim Foster, who showed us their newly installed HEP scheme that enables the centre to be completely self-sufficient in electricity production and reduced their carbon emissions by 80%. Finally, we visited British Geological Survey offices in Keyworth where we were flown over the UK in their 3D visualization room and went to The National Geological Repository where millions of borehole core and specimen samples are stored.

The Blencathra Centre (photo by A.Mijic)

The Blencathra Centre (photo by A.Mijic)

But what I really want to talk about is our last afternoon in Cumbria, when few of us decided to make the most of our stay there and go for a walk up the hills, then down to lakes and back to our base in Keswick. The plan was for Maurice to leave us as close as possible to the beginning of the ascent, from where it should take us approximately four hours of walking to get back. Probably most of coach drivers would just say one big NO when they would see the narrow road he should have taken us through – but not Maurice. With his usual smile and few comments on how there’s no way he’s picking us up anywhere if we get lost, he bravely managed to navigate the bus few hundred meters away from our starting point and left us there. It was 2pm.

A short walk by the lake afterwards, and we were by the starting point of our climb. Just to put things in perspective, I am quite a sporty person. But the view in front of me was not promising – few hundred meters of a very steep climb, completely off road. But there was no way back. So I found a branch that was my best friend during next couple of hours and started climbing. By the time we got to the top of that first hill, my heart was beating like it will jump out of my chest, and my breath was so short that I couldn’t say a word. But I couldn’t stop smiling. Because what followed was one of the most beautiful views I’ve ever seen. And in that middle of nowhere where the only sounds you can hear are the sounds of the howling wind and your pounding heart, you know that all that matters in life are those small moments of happiness that you have to make happen, that hills you have to climb no matter how steep they are; and people who will climb with you, who will follow you and support you, and tell you – come on, you can make it. Who will catch you if you stumble. And you make it. Up the hill, through first project proposal, first lecture with undergrads, first bad decision, first student paper published… you make it through everything.

A hill with a view (photo by K.Weaver)

A team (photo by K.Weaver)

The descent was just a little less difficult, with all the slippery rocks and grass. But the prospect of walking along a flat path kept the pace of our small group, and eventually we ended up in the valley, passing the beautiful lakes. The rest of the walk, which lasted a bit longer than we anticipated, passed in the casual chatting and trying to reach Keswick before the rain and dark. We failed to achieve both, but when we entered our hostel at 7:30pm awfully tired, muddy, sweaty and hungry, the only thing you could see on our faces was a huge grin.

The film ends with a famous quote that happiness (or I would argue any feeling) is not real unless shared. And I couldn’t agree more. As I am writing these closing lines sitting in my study back in London I feel so privileged to be a part of the scientific community that is trying to make a difference in this crazy world; I feel proud of common people like Tony willing to listen and help that community because without their input and support none of our work would make sense; I feel excessively proud of my PhD students that cope with me come rain or shine and do amazing work. Finally, I feel completely elated for two reasons: in those young people traveling with us I saw the yearning for knowledge and appreciation of nature that hopefully will never cease; and I saw the bonding that hopefully will last them for a lifetime. And I share this with you with “a just-in-case mentality” of that happiness’ middle name is maybe and maybe you can feel this too.

MSc Hydrology 2013/14 (photo by J.O'Keeffe)

MSc Hydrology 2013/14 (photo by J.O’Keeffe)

 Acknowledgements

I would like to express my deepest gratitude to all those who have devoted their time to us on this trip: David Macdonald, Bruce Napier, Mike Howe and Chris Jackson from British Geological Survey; Nick Barber from Durham University, Will Cleasby from Eden Rivers Trust and Tony the farmer; friendly Cockermouth resident; and Tim Foster from The Blencathra Centre. Thanks to Wouter Buytaert, Adrian Butler, Deborah Bellaby (University of Lancaster), Judith and Angela for helping me organise this trip; to my MSc students for being such a lovely bunch of people and making the job of taking care of them so easy for me and Jimmy. Special thanks to our driver Maurice who kept us laughing and took us safely through all the narrow roads of Cumbria. And last, but never ever the least, huge thanks to Jimmy for being my huge help and support (especially up those hills!).

Posted in Uncategorized | Leave a comment

Field Work in Uttar Pradesh – First Impressions

India is home to over 1.2 billion people, 200 million of which live in the northern state of Uttar Pradesh. Despite this enormous population, agriculture is the biggest consumer of water in the state, with over 90% of all water used for irrigation.  The remaining 10% is taken up by the domestic, industry and energy sectors. Agricultural water use in India has seen a dramatic rise over the last 60 years, mainly due to the many technical advances heralded by the green revolution, along with the governments’ famous five year plans. Rural electrification has played an important role in providing water access to farmers, particularly in the sections of the Gangetic and Indus basins centred around Punjab, Haryana and Western Uttar Pradesh. This has allowed farmers to become more independent, no longer needing to rely on the intermittent and unreliable supply of water provided by the canals. This has also allowed land further away from the canals catchment to be cultivated. Uttar Pradeshs’ cultivated land has increased dramatically, largely as a result of farmers’ adoption of tube wells as a source of irrigation; a method which, since 1950, has increased from approximately 5 million hectares to almost 14 million in 2010.

p1

Figure 1: Location map of the study area including districts visited during field work

While some information on a state and district level exists, little is known on a smaller scale. The main goal of my field work in India is to collect information on a village and farm scale, and build a dataset on how farmers use their water; where do they get it from, how do they use it and what are the main problems they face in order to irrigate their crops. This information will then be used to quantify the volume of water consumed by agriculture, how this has changed, what the drivers of this change are and importantly to help generate solutions for future water management. This will be greatly helped by developing an understanding of the spatial heterogeneity of water use.

My work has primarily focused on the districts of Sitapur, Jalaun and Hamirpur; all located in the central and south central regions of Uttar Pradesh.  These districts were picked specifically for the sources from which they obtain their irrigation water.  Jalaun is the highest user of canal water in the state, Sitapur is the second highest user of water from tubewells in Uttar Pradesh and one of its biggest grain producers, and according to data  from the States statistical abstract;, farmers in Hamirpur use a relatively even mix of canal, tubewell and other methods to irrigate their crops.

In order to pick the actual locations to visit within each district, a list of villages for each was obtained. Each of the three districts has somewhere between 2000 to 3000 villages and fifteen were randomly picked from each. Within each village a semi-structured interview as conducted with three to five farmers, depending on their availability; in some villages getting five suitable people to interview proved relatively easy, whereas in others trying to meet more than two farmers could be quite a challenge. A semi-structured interview approach was used because it allows a set list of questions to be asked of the interviewees, while also allowing plenty of scope for the farmers to elaborate on their practices and the issues they face. This also allows for the core data collected to be compared between of the farmers interviewed. Questions included the amount of land each farmer had, the type of crops they grow, timing and volume of irrigation water application, water availability and how much it cost them to irrigate. All conversations were audio recorded and transcribed for analysis, all interview locations and where possible the locations of the wells or canals used for irrigation were marked using a GPS.

I thought I had a pretty good idea of what to expect before I headed out; that the southern Bundlekhand district of Jalaun was quite “backward” as the Indians put it, with poorer, less technologically advanced farmers, and Sitapur, just to the north of the State capital Lucknow, with its markets and large population, to be more progressive and wealthy. I expected irrigating your crops to be free or at least cheap and the crops produced by the farmers to be bought by the Government at a guaranteed price.  What I found was almost the complete opposite.

p2

Photo 1: Irrigation of rice field in Sitapur (photo by J. O’Keeffe)

Irrigating crops in these districts can actually be quite expensive. How much depends on your source of water. Irrigation from a tube well usually costs the farmer somewhere in the region of 150 rupees per hour, whether this is done by means of an electric submersible pump or a diesel suction pump. For the farmer who sells the water, the price of using a diesel pump is more or less the same, but is cheaper to use an electric pump, as they usually pay an annual fee of about 25,000 rupees, after which they can use as much as they like; provided there is actually electricity available. Few farmers interviewed had access to electricity for more than eight hours per day, and this was usually at night. As a result, much of the irrigation carried out, particularly in Jalaun was carried out between 4pm- 11pm. If you needed to irrigate your crops outside of these times the only options were diesel, or canal water if available.

 p3

Photo 2: Water abstraction from tubewell using diesel pump in Jalaun (photo by J. O’Keeffe)

A relatively large portion of farmers in the southern district of Jalaun had access to some canal water. Most of this came via a small number of canals supplied by the Matatila Dam and the Betwa River in the district of Jhansi to the south. Canal water provides a much cheaper option than groundwater. While using a tubewell will cost in the region of 2250 rupees per acre for the entire crop of wheat, the same amount of water from a canal will cost about 250 rupees. Access to cheap canal water in Sitapur however is very limited and the vast majority of farmers rely completely on groundwater to irrigate their crops. In addition, while many farmers in Jalaun had access to electric tube wells, few did in Sitapur; the exception being those who could avail of the extremely rare Government tube wells. Irrigation costs comprise a large part of farmers expenditure and with no access to cheap water like users in Jalaun, farmers in Sitapur are less well off and less technically advanced. A clear example of this is the method in which farmers apply their water. While all farmers use flood irrigation, the majority of those in Jalaun employ sprinklers for at least the first application of water, while in Sitapur few farmers even knew what a sprinkler was.

 p4

Photo 3: Road conditions between Jalaun and Hamirpur (photo by S. Swarnkar)

While travelling around Uttar Pradesh and interviewing farmers was certainly interesting and exciting experience, it was not without its challenges. English is an official language in India, but unfortunately for me only 1 farmer out of 109 interviewed could actually converse in it; so all interviews had to be carried out through a translator. Translating from English to Hindi and back again can be tricky, particularly as most locations seem to have their own version of the language. I was very lucky to have had the help of two MTec students; Shirivendra and Somil, who not only could understand my version of English, but also what I was trying to ask.  Both took turns to come out in the field with me along with the fourth member of our team, Moonah the driver. While it’s all well and good to randomly pick villages on a map, actually getting to them turns out to be a bit more complicated. Roads can vary from excellent to terrible, and impassable if there has been a rain shower. Even getting into Hamirpur district proved to be particularly challenging. After almost an entire day of trying, we eventually found what seemed to be the only functioning road into the district.

Overall, it was a great experience in which I collected a lot of what will hopefully turn out to be useful and unique data. I was very lucky to have had lots of help from all at IIT Kanpur, particularly Naresh, my translators Shivendra, Somil , and Moonagh  the driver. Also, Professor Rajiv Sinha who very kindly allowed me to come to Kanpur and provided access to all the resources I needed.

Posted in Uncategorized | Leave a comment

The night of engineering philosophy

That was the topic of the great evening I had with Civil Engineering undergraduates at Imperial College London few nights ago. The idea behind the event was to enhance the communication between students and academic staff through informal chats on even more informal topics. Some wine and beer was served to get the conversation going. That is a completely separate topic, and completely irrelevant for this post, but in the era of interdisciplinary research, it would be interesting to discuss why the initial assumption behind organizing a successful social (academic) event is the presence of alcohol. Not that I mind, on the contrary. Because, interestingly, I just came from the Introductory Evening for the RCUK India Roundtable that was held in the beautiful garden of the Taj Mahal hotel in New Delhi. Only tea and coffee were served, and the conversation was quite vivid. It could be because all the people who were there are supposedly established academics and businessman with fully developed small talk skills. I was in desperate need for a glass of wine.

However. The night of engineering philosophy was organised as a set of roundtables, each with an academic acting as a moderator, and number of students participating. I had a privilege of talking to six very interesting and enthusiastic undergrads – two 1st year and four 3rd students. The first question asked was why we became engineers. And that is in itself a very philosophical question. Why do we become who we are today? What made us choose some profession as our potential future calling at the age of 18? Do we know what we want to become in life when we are 18? Students remembered clearly their reasons for choosing civil engineering: wanting to make something that counts, that you can see and touch; money; not having the slightest idea that there are other engineering disciplines other than civil; seeing it as applied maths. All valid reasons, I guess.

Then I remembered why I became a civil engineer. And it is really a rather peculiar story. I became a civil engineer because I failed to become an architect. That is who I really wanted to become when I was 18. My mother was an architect, so I thought I must have talent. I also thought it is cool to be an almost-artist (my perception of architects then, and I haven’t changed my mind since). And my best friend since we were six wanted to be an architect as well. So, all valid reasons as well. I was preparing for the architecture entrance exam for a year, doing drawing and maths. High school grades were good, the exam came, the exam finished – my best friend passed, I failed. Drawing, of course.

The fall back option not to lose the whole year was going for the closest substitute – civil engineering. Which I strongly believed was just a temporary solution, something I’ll do for a year so that no one can say – oh, there’s the only person from one of the best Grammar schools in Belgrade that didn’t get into the University (… and then in the lower voice – because she never really wanted to admit to herself how bad in drawing she really is). And in all honesty, I am terrible in drawing. One day I will draw something for you and post it here. You can take a look at it when you feel down and it will make your day. It is that bad.

The exam for the Civil Engineering Faculty was just maths test, but bit more difficult maths than for architecture. Although I was always good in maths, I was not that good that I can afford not to practice almost at all before the exam, which is exactly what I did. Or rather what I didn’t do. Fair enough, I was a bit depressed, and I didn’t really care what would happen next. Or I believed that two bad things never happen in sequence. The maths test came, the maths test finished – and I was below the threshold! Only 2.5 out of 100 points, but still below. Which meant that I would be doing nothing for a year, during which I will have time to get properly depressed while wrestling with feelings of self-pity, eventually accept reality for the sake of not losing all my friends who would not be able to bear my moaning any more, and get back to seriously preparing for another exam.

But strange things can happen sometimes. The day after the exam I came back to the University to collect my documents, and at the entrance there was a huge note: “Due to the typo in the question x in the maths test for the Civil Engineering undergraduate studies, all students are given additional 5 points”. And I was in! I became the student at the Civil Engineering Faculty of the University of Belgrade because someone had a bad typing day! But once I was in, I loved every minute of it. Well, not actually every minute, but I felt I belong there. I was actually very good in technical drawing. And I loved finding patterns in the complicated things they tried to teach us. Once you put all the pieces together, and the puzzle is not a puzzle any more, but a clear picture of your solution, then you mind becomes overwhelmed with such a burst of emotions that the feeling becomes almost like an addiction.

Which in my case it probably did. Even now, too many years after I started calling myself a civil engineer, I still have this feeling of butterflies in the stomach every time pieces of a problem puzzle are spread in front of me. Usually, just some of them. Sometimes only a few of them. Then you need to search, and dig, and think, and combine, and think again. Sometimes you have to unlearn something you have learnt to free your mind so that it can reshape a piece that doesn’t fit. And one by one, the pieces are falling into place. In that moment when the picture is complete you understand why you would go through the whole agony again and again and again. I guess my message to all of you young people out there, who may be civil engineers or not, is that even if sometimes the paths we take in life are a combination of an unusual set of circumstances we cannot understand or explain (nor we should try to), we should embrace every opportunity that life gives us, and see if we belong to where that path is taking us. Or to paraphrase the famous saying: you will forget sleepless nights, you will forget miles of running tackled or glasses of wine drunk to ease the mind, but you’ll never forget how you felt when that last piece fitted into the aching void of the puzzle called an engineering problem. I guess this is philosophical enough.

Image

Posted in Uncategorized | Leave a comment