Recent

Author Topic: artificial neural networks | back propagation  (Read 29289 times)

Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: artificial neural networks | back propagation
« Reply #45 on: July 15, 2018, 09:53:16 pm »
Although that is a smart approach it is not a fast approach.
Usually speed is here the foremost importance. String manipulation does not help here.

I don't see how that would make any measurable difference. Most of the time will be spent doing the actual number crunching, not parsing the string containing the expression.

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #46 on: July 26, 2018, 05:43:04 pm »
Now we just need a non-TensorFlow version of MNIST to test against.

I did a MNIST implementation using the CAI framework.

The NN layout is based on one of the CAI CIFAR learning applications, inspired by TF.

Since I'm relatively new in the whole ANN area, I honestly have no idea if I did this correctly. So bear with me. :D

The recognition rate of the test program is pretty good. However, it has major problems recognizing a 9, it always recognizes it as a 3. Don't know why.

You can find everything in this repo. Also a pretrained NN structure and weights file. You still need the CAI framework and the MNIST dataset though.

https://bitbucket.org/108bits/cai-implementations/src/64d3b892d64cb2ca231a7df37ed5d0888ae924c4/MNIST/?at=master

« Last Edit: July 26, 2018, 05:49:52 pm by mw108 »

Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: artificial neural networks | back propagation
« Reply #47 on: July 26, 2018, 07:11:00 pm »
Now we just need a non-TensorFlow version of MNIST to test against.

I did a MNIST implementation using the CAI framework.

The NN layout is based on one of the CAI CIFAR learning applications, inspired by TF.

Maybe I'm not reading your results correctly. But it looks like 20 iterations through the training loop took 2850 seconds. Is that correct? If so, that would seem very slow. MSIST.pas based on TensorFlow took a minute or less depending on system.

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #48 on: July 26, 2018, 07:21:28 pm »
The NN reached 99% accuracy after epoch #13 in ~30 minutes, respectively 99% on the test dataset after epoch #15 and 37min. Just didn't see it earlier to cancel the process. One epoch iterates through all 60.000 MNIST images.

Unfortunately I don't know how TF or your implementation works or what NN layout you used, but learning 60.000 images and reaching 99% accuracy on the 10.000 images training data set all in less than minute seems a bit unrealistic?

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #49 on: July 26, 2018, 07:36:55 pm »
Ok, did some read ups and it seems that a 128 batch indeed completes in approx. 0.5sec in TF.

https://stackoverflow.com/questions/48035125/speed-of-logistic-regression-on-mnist-with-tensorflow

Thats really fast, yea. And CAI very slow, compared to that. But then in the end, it is still WIP. Maybe João can get it faster over time.

And I personally don't understand TF at all. Imho it is very complicated for beginners to get a hold on it.

Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: artificial neural networks | back propagation
« Reply #50 on: July 26, 2018, 08:01:48 pm »
Ok, did some read ups and it seems that a 128 batch indeed completes in approx. 0.5sec in TF.

https://stackoverflow.com/questions/48035125/speed-of-logistic-regression-on-mnist-with-tensorflow

Thats really fast, yea. And CAI very slow, compared to that. But then in the end, it is still WIP. Maybe João can get it faster over time.

And I personally don't understand TF at all. Imho it is very complicated for beginners to get a hold on it.

Yes, TensorFlow-based MNIST.pas 20 iterations through training loop took 12 to 60 seconds, depending on system.

I think TensorFlow is actually easier to understand since it has a lot of related documentation. For example, to understand MNINST.pas or the MNINST.swift program it was based on, see Part 5 here:

https://towardsdatascience.com/machine-learning-with-swift-for-tensorflow-9167df128912?gi=aadadd2fbc78

Obviously speed is important so I don't see any way that I can use CAI. And of course since CAI has a GPL license, I've already had to rule it out. I can't use GPL code in my apps.


schuler

  • Full Member
  • ***
  • Posts: 223
Re: artificial neural networks | back propagation
« Reply #52 on: July 26, 2018, 09:29:40 pm »
mw108,
I'm testing your code now. Deep congrats for your effort. In regards to speed, CAI bench marking with CIFAR10 is impressive. I'll update your code and send it back for your review.

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #53 on: July 26, 2018, 10:29:17 pm »
Great. Looking forward your results. :)

schuler

  • Full Member
  • ***
  • Posts: 223
Re: artificial neural networks | back propagation
« Reply #54 on: July 26, 2018, 10:58:39 pm »
 :) Hello :)
Really enjoyed looking at your code.

My first suggestion is using this NN structure (unorthodox and efficient) :
Code: Pascal  [Select][+][-]
  1.       Input := FNN.AddLayer(TNNetInput.Create(FNNConfig.Width, FNNConfig.Height, FNNConfig.Depth));
  2.       FNN.AddLayer(TNNetConvolutionReLU.Create(16, 5, 0, 1));
  3.       FNN.AddLayer(TNNetMaxPool.Create(4));
  4.       FNN.AddLayer(TNNetConvolutionReLU.Create(64, 3, 1, 1));
  5.       FNN.AddLayer(TNNetConvolutionReLU.Create(64, 3, 1, 1));
  6.       FNN.AddLayer(TNNetFullConnectReLU.Create(64));
  7.       FNN.AddLayer(TNNetFullConnectReLU.Create(32));
  8.       FNN.AddLayer(TNNetFullConnectLinear.Create(FNNConfig.NumClasses));
  9.       FNN.AddLayer(TNNetSoftMax.Create());
  10.  

Then, use a bigger learning rate and smoother learning rate:
Code: Pascal  [Select][+][-]
  1.   LearningRate := 0.001;
  2.   MinLearningRate := 0.00001;
  3.   LearningRateDecay := 0.99;
  4.   StaircaseEpochs := 1;
  5.  

You can compare attachments to your code and apply changes you like. Made the saving of NN less hungry. On bigger machines (such as with 64 and 96 cores), nn saving might be too intensive. I might benchmark along weekend on a high core count computer.

On my dual core notebook, each epoch takes from 50 seconds to 65 seconds (no video card, no opencl).

About benchmarking, we can't compare 3 convolutional layers + 3 FC layers with "simpler methods".

Have a look at how the first epoch goes in the attached image (96% accuracy).

BTW, really well done. Congrats.
« Last Edit: July 26, 2018, 11:08:29 pm by schuler »

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #55 on: July 26, 2018, 11:38:38 pm »
Wow. Thats really a huge improvement! :o Really glad I did something wrong and it wasn't the fault of CAI. :D

On my machine (i7 4GHz, 32GB RAM) one epoch finishes in about 25sec now, with one batch taking about 0.3s (Fwd + Backprop), using no OpenCL.

The test target error rate of 0.5 was reached after the 7th epoch after 166s (~2.5min).

I updated the Repo with the changes. Also added the new pretrained model and the protocol. :)

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #56 on: July 27, 2018, 12:11:01 am »
Small fix in the Repo: Forgot to set the new LearningRateDecay to 0.99. Was still at 0.01.

Result: The test target error rate of <= 0.05 is now already reached almost after the 2nd epoch (0.54). After the 3rd epoch it is 0.34.  :o

mw108

  • New Member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #57 on: July 27, 2018, 09:09:19 am »
Small update to the repo again.

João provided a more reduced NN layout, which finishes a batch in 0.10 to 0.15s and an epoch in ~10s. :)

schuler

  • Full Member
  • ***
  • Posts: 223
Re: artificial neural networks | back propagation
« Reply #58 on: July 27, 2018, 09:23:27 am »
mw108, love seeing your results!

To whom has no idea about what you are talking about, decided to share some links here:
https://en.wikipedia.org/wiki/MNIST_database
https://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html

 

TinyPortal © 2005-2018