Data Cleaning – Or My First Machine Learning Project!


This is the story. I am using the CNN from fastai lesson one to train a neural net to distinguish between mala beads and rosary beads. I feel like this is really keeping with my whole conscious computation MO.  Mala bracelets are used for counting mantras. They include 108 beads plus a guru bead. Rosary beads are beads that let you count your prayers! No so different. But they look a bit different. Rosary beads have a cross, mala beads dont. Rosaries are held differently and are not as long. I am sure there are other differences – that perhaps an AI neural net can discover.  Then perhaps the neural net can create a new kid of prayer ritual object!

So, back when I lived in Hong Kong modeling yield curves, the guy next to me had a phd in economics AND statistics. He spent his entire day cleaning data. Today I had a small taste of that –  cleaning up my images for use with fast ai.

After much struggle, I finally installed fastai on the community heap cluster. I am almost done with my terraform script, the subject of the next blog post.   However no soon had I gotten everything up then I got the same error I got on paperspace!

 im = cv2.imread(str(fn), flags).astype(np.float32)/255

My images looked correct on jupyter but then I ran a cell to try and convert each jpg to cv2. Guess what – not a single one worked. Something was wrong with my images!


There must be something wrong with my jpgs – lets do some sort of conversion.  Actually I did not have to do that.  When,  I included the path file with the filename in cv2.imread  it was very trivial finding the offending file: download10.jpg.  Keep that code kids, it is very useful.

Once it was removed,  my model ran! Although the scores were bad and it scored the  mala beads not rosaries.  rank

To improve this I am going to add more images and then play with learn fit, then stick it online maybe I’ll use glitch.

Now I am going to go encounter for the millionth time the reality of experience and to forge in the smithy of my soul the uncreated conscious of my race.


Leave a Reply