A new project

As you have certainly seen on my website: businesshackers.com I am going to trade eur/usd using NN... If you would like to discuss it please contact me on skype: darioaarhus

Take care

Site recommendation Fast Artificial Neural Network Library

Fast Artificial Neural Network Library is a free open source neural network library, which implements multilayer artificial neural networks in C with support for both fully connected and sparsely connected networks. Cross-platform execution in both fixed and floating point are supported. It includes a framework for easy handling of training data sets. It is easy to use, versatile, well documented, and fast. PHP, C++, .NET, Ada, Python, Delphi, Octave, Ruby, Pure Data and Mathematica bindings are available. A reference manual accompanies the library with examples and recommendations on how to use the library. A graphical user interface is also available for the library.

link

The discussion board is now on

Ok guys... no we are ready to post our news and posts on the discussion board... it is absolutely vital that you share your experience with other users of th d.b. The more we know, the more successfull we will be...

Link: click here

Announcement

I will soon create a discussion board where we will be able to discuss our experiences... the aim of the d.b. will be to share our knowledge about NN and put it into practice.. please feel free to make any suggestions... adding comments...

p.s. the discussion board is now available at http://neuraldiscussions.ning.com/

references: different applications of NN

recently one of the readers of this blog asked me for references... those references are for those looking into the applications of NN... i hope you will find them useful...

Blair, A.D. (1997), "Scaling Up RAAMs," Brandeis University Computer Science Technical Report CS-97-192, http://www.demo.cs.brandeis.edu/papers/long.html#sur97

Blum, A., and Rivest, R.L. (1989), "Training a 3-node neural network is NP-complete," in Touretzky, D.S. (ed.), Advances in Neural Information Processing Systems 1, San Mateo, CA: Morgan Kaufmann, 494-501.

Chalmers, D.J. (1990), "Syntactic Transformations on Distributed Representations," Connection Science, 2, 53-62, http://ling.ucsc.edu/~chalmers/papers/transformations.ps

Chalmers, D.J. (1996), The Conscious Mind: In Search of a Fundamental Theory, NY: Oxford University Press.

Chrisman, L. (1991), "Learning Recursive Distributed Representations for Holistic Computation", Connection Science, 3, 345-366, ftp://reports.adm.cs.cmu.edu/usr/anon/1991/CMU-CS-91-154.ps

Collier, R. (1994), "An historical overview of natural language processing systems that learn," Artificial Intelligence Review, 8(1), ??-??.

Devroye, L., Györfi, L., and Lugosi, G. (1996), A Probabilistic Theory of Pattern Recognition, NY: Springer.

Faragó, A. and Lugosi, G. (1993), "Strong Universal Consistency of Neural Network Classifiers," IEEE Transactions on Information Theory, 39, 1146-1151.

Hadley, R.F. (1999), "Cognition and the computational power of connectionist networks," http://www.cs.sfu.ca/~hadley/online.html

Hammerton, J.A. (1998), "Holistic Computation: Reconstructing a muddled concept," Connection Science, 10, 3-19, http://www.tardis.ed.ac.uk/~james/CNLP/holcomp.ps.gz

Judd, J.S. (1990), Neural Network Design and the Complexity of Learning, Cambridge, MA: The MIT Press.

Lugosi, G., and Zeger, K. (1995), "Nonparametric Estimation via Empirical Risk Minimization," IEEE Transactions on Information Theory, 41, 677-678.

Orponen, P. (2000), "An overview of the computational power of recurrent neural networks," Finnish AI Conference, Helsinki, http://www.math.jyu.fi/~orponen/papers/rnncomp.ps

Plate, T.A. (1994), Distributed Representations and Nested Compositional Structure, Ph.D. Thesis, University of Toronto, ftp://ftp.cs.utoronto.ca/pub/tap/

Pollack, J. B. (1990), "Recursive Distributed Representations," Artificial Intelligence 46, 1, 77-105, http://www.demo.cs.brandeis.edu/papers/long.html#raam

Siegelmann, H.T. (1998), Neural Networks and Analog Computation: Beyond the Turing Limit, Boston: Birkhauser, ISBN 0-8176-3949-7, http://iew3.technion.ac.il:8080/Home/Users/iehava/book/

Siegelmann, H.T., and Sontag, E.D. (1999), "Turing Computability with Neural Networks," Applied Mathematics Letters, 4, 77-80.

Sima, J., and Orponen, P. (2001), "Computing with continuous-time Liapunov systems," 33rd ACM STOC, http://www.math.jyu.fi/~orponen/papers/liapcomp.ps

Valiant, L. (1988), "Functionality in Neural Nets," Learning and Knowledge Acquisition, Proc. AAAI, 629-634.

White, H. (1990), "Connectionist Nonparametric Regression: Multilayer Feedforward Networks Can Learn Arbitrary Mappings," Neural Networks, 3, 535-550. Reprinted in White (1992b).

White, H. (1992a), "Nonparametric Estimation of Conditional Quantiles Using Neural Networks," in Page, C. and Le Page, R. (eds.), Proceedings of the 23rd Sympsium on the Interface: Computing Science and Statistics, Alexandria, VA: American Statistical Association, pp. 190-199. Reprinted in White (1992b).

White, H. (1992b), Artificial Neural Networks: Approximation and Learning Theory, Blackwell.

White, H., and Gallant, A.R. (1992), "On Learning the Derivatives of an Unknown Mapping with Multilayer Feedforward Networks," Neural Networks, 5, 129-138. Reprinted in White (1992b).

sources

have a look at this sources:

* Neural Network Toolbox for MATLAB Open this result in new window
www.mathworks.com/products/neuralnet
* Neural Nets Open this result in new window
Lecture notes from an MSc course. Covers TLUs, delta rule, multilayer nets, Hopfield nets, Kohonen nets, node types, cubic nodes.
www.shef.ac.uk/psychology/gurney/notes/contents.html
* Neural Networks at your Fingertips Open this result in new window
Neural network simulators for eight different network architectures with embedded example applications coded in portable ANSI C.
www.neural-networks-at-your-fingertips.com
* Neural Network Using Genetic Algorithms (NNUGA) Open this result in new window
Includes screen shots and the free source code.
www.cs.bgu.ac.il/~omri/NNUGA
* Neural Machines Open this result in new window
Discusses the creation and application of advanced neural network technology in optimization and pattern recognition.
www.neuralmachines.com
* Perceptron, The Open this result in new window
A small and free neural network software package, demonstrating using NN's for pattern classification. Includes screen shots and the source code.
www.cs.bgu.ac.il/~omri/Perceptron
* Neural Network Announcements and General Information Open this result in new window
At Los Alamos.
www.www-xdiv.lanl.gov/XCM/neural/neural_announcements.html
* Web Directory: Sumeet's Neural Net Links Open this result in new window
www.geocities.com/CapeCanaveral/Lab/3765/neural.html
* FAQ - comp.ai.neural-nets Open this result in new window
www.cs.cmu.edu/Groups/AI/html/faqs/ai/neural/faq.html
* Joone: Java Object Oriented Neural Engine Open this result in new window
Neural net framework to create, train, and test neural networks.
www.joone.org
* Web Directory: Neural Networks Open this result in new window
Listing newsgroups, list servers, and mailing lists.
www.emsl.pnl.gov:2080/docs/cie/neural/newsgroups.html
* LANL Advanced Adaptive Control Open this result in new window
Neural networks adaptive control applications.
www.www-xdiv.lanl.gov/XCM/neural/projects/projects.html
* FAQ - Neural Networks Open this result in new window
www.ftp://ftp.sas.com/pub/neural/FAQ.html

Network Architectures

Supervised Networks

Supervised neural networks are trained to produce desired outputs in response to sample inputs, making them particularly well suited to modeling and controlling dynamic systems, classifying noisy data, and predicting future events.

* Feedforward networks have one-way connections from input to output layers. They are most commonly used for prediction, pattern recognition, and nonlinear function fitting. Supported feedforward networks include feedforward backpropagation, cascade-forward backpropagation, feedforward input-delay backpropagation, linear, and perceptron networks.
* Radial basis networks provide an alternative, fast method for designing nonlinear feedforward networks. Supported variations include generalized regression and probabilistic neural networks.
* Dynamic networks use memory and recurrent feedback connections to recognize spatial and temporal patterns in data. They are commonly used for time-series prediction, nonlinear dynamic system modeling, and control system applications. Prebuilt dynamic networks in the toolbox include focused and distributed time-delay, nonlinear autoregressive (NARX), layer-recurrent, Elman, and Hopfield networks. The toolbox also supports dynamic training of custom networks with arbitrary connections.
* LVQ is a powerful method for classifying patterns that are not linearly separable. LVQ lets you specify class boundaries and the granularity of classification.

Unsupervised Networks

Unsupervised neural networks are trained by letting the network continually adjust itself to new inputs. They find relationships within data and can automatically define classification schemes.

The Neural Network Toolbox supports two types of self-organizing, unsupervised networks: competitive layers and self-organizing maps.

Competitive layers recognize and group similar input vectors. By using these groups, the network automatically sorts the inputs into categories.

Self-organizing maps learn to classify input vectors according to similarity. Unlike competitive layers they also preserve the topology of the input vectors, assigning nearby inputs to nearby categories.

(source: http://www.mathworks.com/products/neuralnet/descripton3.html)

Comments about the inputs

Some of my readers asked me about the inputs that worked best for applying NN to the eur/usd. The answer would be:
--> raw prices OHCL performed poor in general even if lags were introduced
--> technical indicators enhanced the explanatory power of the models ( the best result so far has been 84% trend accuracy)
--> financial ratios, we need to try to use them; a recent article for example included 24 ratios about the Canadian stock market

NN results...

Someone asked me to post the results... i want to give you an example of the potential profitability of NN if applied correctly... look at my post on moneytec:

link to the results

Reply to Zendini comments

Thanks a lot for you comment Zendini.. i really appreciate the fact that you decided to express your view on the subject...I still believe that NN have huge potential... i have recently read dozens of papers about empirical results starting from 1988... and i'm doing research on s&p500 data with very good results... I have also written a paper for a university project that i can't, unfortunately post on the blog for obvious reasons... so if you have doubts about the subject please refere to the following sources:

J. Loofbourrow & T. Loofbourrow, “Neural Networks for Trading”, AI in Finance (Special Report), 42-51
J. Zirilli, Financial Prediction Using Neural Networks, London, 1997, International Thompson Computer Press
A. Skabar & I. Cloete, Neural Networks, Financial Trading and the Efficient Markets Hypothesis, XXV Australasian Computer Science Conference 2002, Australian Computer Society, Inc.
Shaikh A. Hamid & Z. Iqbal, Using neural networks for forecasting volatility of S&P 500 Index futures prices, Journal of Business Research 57 (2004) 1116-1125.
K. Kohara, T.Ishikawa, Y.Fukuhara, Yukihiro Nakamura, Stock Price Prediction Using Prior Knowledge and Neural Networks, Intelligent Systems in Accounting, Finance and Management vol 6: 11-22, 1997
J. Kamruzzaman & R. Sarker, Comparing ANN Based Models with ARIMA for Prediction of Forex Rates.
G. E. P. Box and G. M. Jenkins, Time Series Analysis: Forecasting and Control Holden-Day, San Francisco, CA
G. Zhang and M.Y. Hu, Neural Network Forecasting of the British Pound/USD Dollar Exchange Rate, “OMEGA; Int. Journal of Management Science, 26, pp 495-506, 1998
G. Deboeck, Trading on the Edge: Neural, Genetic and Fuzzy Systems for Chaotic Financial Markets
An-Sing Chen, Hazem Daouk, Mark T. Leung, Application of Neural Networks to an Emerging Financial Market: Forecasting and Trading the Taiwan Stock Index 2001
Wei Cheng, Lorry Wagner, Chien-Hua Lin, Forecasting the 30-year U.S. Treasury Bond with a System of Neural Networks,NeuroVe$t Journal, January/February 1996
S. Dutta and S. Sheckhar, Bond Rating: A Non-Conservative Application of Neural Networks, Proceedings of the IEEE International Conference on Neural Networks,1998, 2, 443-450
J. Moody and J. Utans, Architecture Selection Strategies for Neural Networks, In. A. P. Refenes, Ed., Neural Networks in the Capital Markets, Chichester, England, 1995, John Wiley & Sons, 277-300.
T. Lubecke, Kyung Doo Nam, R. Markland, C. Kwok, Combining Foreign Exchange Forecasts using Neural Networks, 1998, Global Finance Journal 9(1): 5-27
A. Fadlalla & Chien-Hua Lin, An Analysis of the Applications of Neural Networks in Finance, 2001, Interfaces 31: 4 July-August 2001 (pp. 112-122), Informs.

Have a nice day!

Discussion board...

I'm thinking of creating a discussion board about NN and their application to financial markets... even though i got great feedback from readers of this blog i think we could try to discuss the issue together more often... could you please comment on this..
take care...

Limits and application of NN

"In principle, NNs can compute any computable function, i.e., they can do everything a normal digital computer can do (Valiant, 1988; Siegelmann and Sontag, 1999; Orponen, 2000; Sima and Orponen, 2001), or perhaps even more, under some assumptions of doubtful practicality (see Siegelmann, 1998, but also Hadley, 1999).

Practical applications of NNs most often employ supervised learning. For supervised learning, you must provide training data that includes both the input and the desired result (the target value). After successful training, you can present input data alone to the NN (that is, input data without the desired result), and the NN will compute an output value that approximates the desired result. However, for training to be successful, you may need lots of training data and lots of computer time to do the training. In many applications, such as image and text processing, you will have to do a lot of work to select appropriate input data and to code the data as numeric values.

In practice, NNs are especially useful for classification and function approximation/mapping problems which are tolerant of some imprecision, which have lots of training data available, but to which hard and fast rules (such as those that might be used in an expert system) cannot easily be applied. Almost any finite-dimensional vector function on a compact set can be approximated to arbitrary precision by feedforward NNs (which are the type most often used in practical applications) if you have enough data and enough computing resources.

To be somewhat more precise, feedforward networks with a single hidden layer and trained by least-squares are statistically consistent estimators of arbitrary square-integrable regression functions under certain practically-satisfiable assumptions regarding sampling, target noise, number of hidden units, size of weights, and form of hidden-unit activation function (White, 1990). Such networks can also be trained as statistically consistent estimators of derivatives of regression functions (White and Gallant, 1992) and quantiles of the conditional noise distribution (White, 1992a). Feedforward networks with a single hidden layer using threshold or sigmoid activation functions are universally consistent estimators of binary classifications (Faragó and Lugosi, 1993; Lugosi and Zeger 1995; Devroye, Györfi, and Lugosi, 1996) under similar assumptions. Note that these results are stronger than the universal approximation theorems that merely show the existence of weights for arbitrarily accurate approximations, without demonstrating that such weights can be obtained by learning.

Unfortunately, the above consistency results depend on one impractical assumption: that the networks are trained by an error (L_p error or misclassification rate) minimization technique that comes arbitrarily close to the global minimum. Such minimization is computationally intractable except in small or simple problems (Blum and Rivest, 1989; Judd, 1990). In practice, however, you can usually get good results without doing a full-blown global optimization; e.g., using multiple (say, 10 to 1000) random weight initializations is usually sufficient.

One example of a function that a typical neural net cannot learn is Y=1/X on the open interval (0,1). An open interval is not a compact set. With any bounded output activation function, the error will get arbitrarily large as the input approaches zero. Of course, you could make the output activation function a reciprocal function and easily get a perfect fit, but neural networks are most often used in situations where you do not have enough prior knowledge to set the activation function in such a clever way. There are also many other important problems that are so difficult that a neural network will be unable to learn them without memorizing the entire training set, such as:

* Predicting random or pseudo-random numbers.
* Factoring large integers.
* Determing whether a large integer is prime or composite.
* Decrypting anything encrypted by a good algorithm.

And it is important to understand that there are no methods for training NNs that can magically create information that is not contained in the training data.

Feedforward NNs are restricted to finite-dimensional input and output spaces. Recurrent NNs can in theory process arbitrarily long strings of numbers or symbols. But training recurrent NNs has posed much more serious practical difficulties than training feedforward networks. NNs are, at least today, difficult to apply successfully to problems that concern manipulation of symbols and rules, but much research is being done.

There have been attempts to pack recursive structures into finite-dimensional real vectors (Blair, 1997; Pollack, 1990; Chalmers, 1990; Chrisman, 1991; Plate, 1994; Hammerton, 1998). Obviously, finite precision limits how far the recursion can go (Hadley, 1999). The practicality of such methods is open to debate.

As for simulating human consciousness and emotion, that's still in the realm of science fiction. Consciousness is still one of the world's great mysteries. Artificial NNs may be useful for modeling some aspects of or prerequisites for consciousness, such as perception and cognition, but ANNs provide no insight so far into what Chalmers (1996, p. xi) calls the "hard problem":

Many books and articles on consciousness have appeared in the past few years, and one might think we are making progress. But on a closer look, most of this work leaves the hardest problems about consciousness untouched. Often, such work addresses what might be called the "easy problems" of consciousness: How does the brain process environmental stimulation? How does it integrate information? How do we produce reports on internal states? These are important questions, but to answer them is not to solve the hard problem: Why is all this processing accompanied by an experienced inner life?"
(source)

Book about NN by Smith

Today i will start reading a new book by Murray Smith called "NN for statistical modeling". The book is about backpropagation, which is supposed to be the most widely used and studied method in the NN community.The book was published in 1996 and seems to be a rather understandable one combining theoretical and practical issues. I will keep you informed about it.

A second application of NN

The HR-2 robot was constructed during a period of three months at Chalmers University in Sweden. It has 22 degrees of freedom which enables it to easily move around imitating human motions. The robot is also equipped with stereovision giving it possibilities to perform hand-eye coordination. For that task an artificial neural network is evolved. Furthermore, the artificial brain is capable of tracking faces as well as recognising them. The HR-2 is also able to speak.

A non financial application of NN

This simulated car is controlled by a neural network that has been trained by evolutionary algorithms (aka genetic algorithms). Its input are several rangefinder sensors, and outputs are speed and steering commands. No human knowledge has gone into designing the driving behaviour.

John Kenneth Galbraith died

"John Kenneth Galbraith, the iconoclastic economist, teacher and diplomat and an unapologetically liberal member of the political and academic establishment that he needled in prolific writings for more than half a century, died yesterday at a hospital in Cambridge, Mass. He was 97.

Galbraith lived in Cambridge and at an "unfarmed farm" near Newfane, Vt. His death was confirmed by his son J. Alan Galbraith.

Galbraith was one of the most widely read authors in the history of economics; among his 33 books was "The Affluent Society" (1958), one of those rare works that forces a nation to re-examine its values. He wrote fluidly, even on complex topics, and many of his compelling phrases - among them "the affluent society," "conventional wisdom" and "countervailing power" - became part of the language. An imposing presence, lanky and angular at 6 feet 8 inches tall, Galbraith was consulted frequently by national leaders, and he gave advice freely, though it may have been ignored as often as it was taken. Galbraith clearly preferred taking issue with the conventional wisdom he distrusted".
(source)

Help us to promote the site

If you like this blog please add it to:
http://del.icio.us/

or link it from other sites...
thanks a lot

ANN and ARIMA article summary

I would like to draw your attention to a paper published on the ASOR Bulletin in 2003. It's titled "comparing ANN Based Models with ARIMA for Prediction of Forex Rates". It's an interesting paper written in plain language. Let me give you a summary of the main ideas:
- the ANN models used (including SB, SCG and BR)all outperformed the ARIMA models (Box-Jenkins);
- the analysis was about the forex market, six currency pairs in total;
- "ANN proved to be very effective in describing the dynamics of non-stationary time series due to its unique non-parametric, non-assumable, noise-tolerant, and adaptive approximators that can map any nonlinear function without a priori assumptions about the data";
- details about ARIMA are available in Jarret "Business Forecasting Methods";
- the idea of NN is explained;
- multilayer feedforward network (MFN) is described briefly;
- no study has been reported to analytically determine the generalization performance of each algorithm;
- the author uses MA5, MA10, MA20, MA60, MA120 (Moving averages of given lengths);
- the NN used had 6 inputs, 1 layer and one output;
- the statistical metrics used to evaluate the performance are: Mean Square Error, Mean Absolute Error, Directional Symmetry;
- neural networks models produce much better performance than the conventional ARIMA models for both shorter and longer term forecasting;

A link

Good morning,

i would like to provide all of you who decided to join this attempt to understand NN with an interesting link. It's not directly connected with the subject of the blog, but it's rather interesting:
The pleasure of finding things out - RP Feynman --> link

Learning paradigms

we can distinguish three main learning paradigms:
- supervised learning
- nonsupervised learning
- reinforcement learning

Artificial intelligence vs. cognitive modelling

"Artificial intelligence and cognitive modeling try to simulate some properties of neural networks. While similar in their techniques, the former has the aim of solving particular tasks, while the latter aims to build mathematical models of biological neural systems.

In the artificial intelligence field, artificial neural networks have been applied successfully to speech recognition, image analysis and adaptive control, in order to construct software agents (in computer and video games) or autonomous robots. Most of the currently employed artificial neural networks for artificial intelligence are based on statistical estimation, optimisation and control theory.

The cognitive modelling field is the physical or mathematical modelling of the behaviour of neural systems; ranging from the individual neural level (e.g. modelling the spike response curves of neurons to a stimulus), through the neural cluster level (e.g. modelling the release and effects of dopamine in the basal ganglia) to the complete organism (e.g. behavioural modelling of the organism's response to stimuli)".
(source)

NN definition

Let's start by some basic issues... let's define neural networks:

" A neural network is an interconnected group of biological neurons. In modern usage the term can also refer to artificial neural networks, which are constituted of artificial neurons. Thus the term 'Neural Network' specifies two distinct concepts:

1. A biological neural network is a plexus of connected or functionally related neurons in the peripheral nervous system or the central nervous system. In the field of neuroscience, it most often refers to a group of neurons from a nervous system that are suited for laboratory analysis.
2. Artificial neural networks were designed to model some properties of biological neural networks, though most of the applications are of technical nature as opposed to cognitive models."

(source: http://en.wikipedia.org/wiki/Neural_networks ).

Introduction to the blog

Welcome!

This is a blog that will allow me to introduce the idea of NN and their application for financial markets... this mix will represent a mix of theory, empirical findings, forecasts and opionions... it will cover the topic from a newbie to a professional level... you are welcome to add comments and write anything you want on the topic...