February 02, 2021

Updating to FastAI v2 on ubuntu and anaconda

Follow-up to Keras, TensorFlow and more on the SC–RTP desktops from Rudo's blog

I wanted to upgrade my FastAI installation to V2, using my existing ubuntu installation of anaconda. After a few wrong tipps on google, I converged to the following.

First, I made a new conda environment, i.e.

$ conda create -n FastAI2

Then I installed all that was needed into the new environment:

$ conda update -n base -c defaults conda
$ conda install -c fastai -c pytorch fastai
$ conda install jupyter
$ conda install -c fastai nbdev

OK, that seems to work, i.e. the example jupyter notebooks from the FastAI V2 course now run.

January 16, 2021

WebDAV access to UoW H+M drives

In the Ubuntu (20.04) default file manager, if you type in the "Connect to Server" box

davs://myfiles.warwick.ac.uk/hcwebdav

then I get a username/password prompt. Logging in with ITS credentials, I then have full access to the centrally managed H+M drives. In order to make the access more permanent, I "bookmark" the connection in nautilus.


October 23, 2020

Keras, TensorFlow and more on the SC–RTP desktops

Follow-up to Using PyTorch on the SC–RTP machines from Rudo's blog

This set of commands allows to load a working Keras/TF/scikit ML environment which will run on a GPU if present

$ module load GCC/8.3.0  CUDA/10.1.243  OpenMPI/3.1.4
$ module load IPython/7.9.0-Python-3.7.4
$ module load TensorFlow/2.1.0-Python-3.7.4 
$ module load Keras
$ module load scikit-learn
$ jupyter notebook &

using the versions

tensorflow:  2.1.0 , keras:  2.3.1 sklearn:  0.21.3

October 16, 2020

Using PyTorch on the SC–RTP machines

Follow-up to Installing your own TensorFlow/Kears + FastAI on Ubuntu from Rudo's blog

We (the DisQS team) have now moved back from Cergy to the University of Warwick where the Scientific Computing RTP provides a centrally managed Linux desktop setup as well as bespoke HPC solutions, including some GPU-enabled machines. This means that we need to switch from our self-managed Ubuntu/Anaconda solution to the managed SC-RTP desktop. The following lines, executed in an SC-RTP terminal, allow us to use PyTorch, or preferred ML package at the moment, as follows

module load GCC/8.3.0  OpenMPI/3.1.4
module load IPython/7.9.0-Python-3.7.4
module load PyTorch
module load scikit-learn
pip install --user torchvision==0.4.0
module save ML (or some such name)

and later one can retrieve these settings via

module restore ML

Please not that we need to install torchvision locally via pip as it's not available on the SC-RTP system as a module. The pip mechanism is what replaces the "conda install ..." used under anaconda for missing libraries. Also, we found that the newer torchvision version 0.7.0 gave some error with our codes, so we are using an older version that seems to work. This may not be needed for your code. Obviously, the use of "module save/restore" is optional, but very convenient.

jupyter notebook &

then works as before/desired.


March 31, 2020

Screen casting while working from home

Writing about web page https://www.howtogeek.com/446706/how-to-create-a-screencast-on-linux/

As I am sitting at home due to the Covid-19 pandemic and the closure of "outside life", I find myself having to give feedback to PhD students who are not around. Normally, I would do this with them sitting next to me. Now, I am trying to do it using a "Screencast", i.e. a video that shows my screen and records my voice, perhaps even my video, when I go through their work and tell them what I think about it and how it could be improved. I have found that under Linux, there is a varirty of useful and free such screencast software. After a bit of googling, I found

https://www.howtogeek.com/446706/how-to-create-a-screencast-on-linux/

and at the moment, I am using the "recordmyDesktop" tool which one can install via the ususal Linux software channels. After a few tries, I think this is indeed a great tool. I'll let you know what the students think ...


March 05, 2020

list of submitted projects and information about late submission penalties

Follow-up to Submission date and instructions for your project work from Rudo's blog

Dear all,

thus far I have received submissions from (10/02/30 10:00)

  1. Jettawat Khantaveramongkol
  2. Juan Roman Roche
  3. Hendrik Schlueter
  4. Alexios Malaquias
  5. DanielPepin Fowan
  6. Burak Civitcioglu
  7. Roman Perrier (one day late)
  8. Jeremy Cadran (two days late)

Please note that I intend to apply late submission penalties, following roughly similar Warwick penalties which penalize a late submission with a 5% reduction in final points for every day late. A late day starts in the first minute after the dealine and the time of submission follows from the time of the last upload.

Regards,

-Rudo


March 02, 2020

PS: don't worry too much about maximizing epochs

Follow-up to Submission date and instructions for your project work from Rudo's blog

Just to add: do not worry too much about running your code to the maximum of epochs. It is clear to us that most people do not have access to state of the art graphics cards or hundreds of processors. What is more important is to see that you came up with a good idea where machine learning can be used, executed the right commands to get some training and analysis done, and found the right data to train on. In fact, we migth even try to run some of your code for more epochs to see how results can get better!


Submission date and instructions for your project work

Follow-up to teaching schedule, now ready from Rudo's blog

Dear all,

you should now all be close to finishing your project work for the machine learning course. I hope it has all gone well and that you had some fun exploring machine learning. The official date for submission is

Friday, March 6 at 17:00.

For submission, please remember that we want you to submit a running machine learning code using a jupyter notebook with comments that explain step by step what you did and why. Please also supply the data so that Djena and I can run the code ourselves. The easiest way to send us the submission is via

https://files.warwick.ac.uk/rroemer/sendto

where you can upload both jupyter notebook and data (preferably as a single .zip file). If your data files are too big for upload there (I have 4.5GB in total on that upload drive), please send a link at which we can then download the data. Please remember that it is your responsibility to make sure data and notebook work "out of the box", i.e. without us having to change anything before they work, including directories.

We aim to have feedback for you by the end of March and will communicate this by email to each of you (so please also let me have your preferred email address in your jupyter notebook).

Cheers!


December 12, 2019

teaching schedule, now ready

Follow-up to Machine Learning – a hands–on course from Rudo's blog

Dear all,

the schedule of the course has now become clearer: we continue to teach every Monday 0930-1230 in room E4.28 (except for one day in January and one day in February) for 10 weeks overall. For the 1st 5 weeks (with 4 more weeks to come), I will actually "teach", i.e. with prepared lectures for 2 hours 0930-1130, while Djenabou will be available for 1 hr more 1130-1230 to help you with the exercises. Then in the last 5 weeks, Djenabou will be available 0930-1230 to help with your project work (she won't do it for you, but might be able to help with technical problems).

A full schedule should be available soon.

See you next Monday!

-Rudo


December 09, 2019

Thoughts on the 1st lecture session

Follow-up to Installing your own TensorFlow/Kears + FastAI on Ubuntu from Rudo's blog

Dear all,

[1] today in the lecture, we saw that it was not entirely simple to install your own ML machine with Tensorflow/Keras and FastAI. I have now made a blog entry on how to do this. You can find it here.

[2] In addition, when running the FastAI code, we saw that it might take a long time even for just the pets categorization. I just ran the pets notebook (please download a new version) and have made the pets-stage-1 and pets-stage-1-50 trained models available on the download area. You need to move these models into where FastAI keeps them which in my set-up is

/home/phsht/.fastai/data/oxford-iiit-pet/images/models

Just copy the twp .pth files there and you should be able to run the pets code after the

learn.load('pets-stage-1')

fastai commands in the notebook.

[3] When you "play" with the notebook called "FastAI-lesson2-download.ipynb", note that the file data.zip contains the cats/teddys/wolves that I managed to download and use for the exercise.

[4] Some of you argued that your own machines may not be able to do meaningful ML as you do not have a GPU. Have a look at what the FastAI guys have collected in terms of possible free GPU instances, i.e. Using a GPU. I have not yet used any of these services, but would be keen to learn from your experiences!

OK, keep checking this blog for up-to-date information before we meet again next week.


June 2021

Mo Tu We Th Fr Sa Su
May |  Today  |
   1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30            

Search this blog

Tags

Galleries

Most recent comments

  • Please note that this link to download is now obsolete. Attendees for the 2021 MPAGS/CYU course shou… by Rudo Roemer on this entry
  • To load Tensorflow and PyTorch (and Keras) modules on ORAC may you may have to module load a few mor… by Daniel Paget on this entry
  • That's cool. Perhaps you want to show people during class next week how to run on Kaggle? by Rudo Roemer on this entry
  • Regarding possible alternatives to benefit form a GPU, I have personally used Kaggle, by Google, its… by Juan on this entry
  • Hello to you all, I was just trying to make the programs run in my VirtualBox with Ubuntu 18.04 inst… by Malaquias Correa on this entry

Blog archive

Loading…
RSS2.0 Atom
Not signed in
Sign in

Powered by BlogBuilder
© MMXXI