JPH06203005A - High-speed segmented neural network and method for constructing the same - Google Patents

Abstract

(57) [Abstract] [Purpose] To provide a neural network of a feed-forward control system and a neural network that is high-speed operation and its construction technology. [Structure] The present segmented network 10 is composed of a plurality of network layers 1 to 5 stacked in an upward pyramid shape. Each of these network layers 1 to 5 is composed of a plurality of sub-networks SN12, a plurality of nodes are arranged in each of the sub-networks SN12, and the nodes are completely or partially interconnected layered neural network array. Configure within. The input / output of each sub-network SN12 is a 1-bit digital value limited to “0” or “1”. Each sub-network S
N12 is independent of all other sub-networks SN12 in a given network layer 1-5, thus partitioning each network layer 1-5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer processing system, and more particularly to a high speed neural network and its construction technology.

[0002]

2. Description of the Related Art An artificial neural network (neural network) is a large-scale group of parallel neuron-type elements that are interconnected by a specific method, and the method is not limited to the following. , Optical character recognition, pattern recognition, machine learning, process control and voice recognition. The most common structure in this neural network system is a network of non-linear processing elements, the "nodes" of which are associated with multiple inputs via information processing channels or "weights". Each node can handle its many inputs and weights,
It outputs one signal each. This network often has multiple layers, with each layer other than the first one receiving the output of the previous layer as input, and the last layer of the network generally providing the output stimulus.

Neural networks are capable of simulating biological neural tissue features at a very basic level. This biological neural tissue has many advantages, including the following: the ability to adapt to, control, and process a wide range of environments, effectively in real time. The ability to operate in the form of massively parallel processing to function, the ability to tolerate defects, the ability to handle internal errors in the circuitry itself, and the ability to learn by example. In neural networks, pre-training is necessary to get useful results. However, in many applications, it is sufficient to perform batch backpropagation training once. Once trained, the resulting "weights" are saved and the weights are modified for subsequent use during non-training test mode or "forward mode" operation.

In the most highly interconnected networks, the number of weights increases non-linearly as the number of nodes and the number of inputs increases linearly. For example, if the number of nodes increases linearly with the number of inputs in a single layer of a fully interconnected network, the number of weights will increase in proportion to the square of the number of inputs.

More specifically, for example, the number of inputs is 10,
Within a fully interconnected small network layer with 10 nodes, 100 weight numbers would be needed. However, for 1,000 inputs and 1,000 nodes, the required weight number is 1,000,000, which is a tremendous amount. This not only entails enormous hardware required for simulation, but also makes it extremely complicated, and when a single processing system is used, it becomes slow when the forward control method is used.

[0006]

The operation of the real-time network in the forward control system (for example, a pattern recognition system using a neural network) is equivalent to the operation when using ultrafast simple hardware. , This is often difficult to achieve. Because
This is because applying a large number of pattern recognitions requires a large number of inputs and / or nodes. Many pattern recognition systems developed to date use simple high speed hardware systems. Furthermore, if the number of inputs is extremely large, real parallel hardware realization of a neural network is impossible in reality. In many real-time pattern recognition applications, neural networks are required to have interconnect speeds of billions of times or more per second. However, neural networks that operate at such effective speeds are not currently available. Furthermore, most of the pattern recognition neural networks that have been developed so far do not have a structure that enables high-speed pattern recognition by hardware.

Therefore, what is needed is a neural network system technique that adapts itself to implementations with large numbers of inputs and nodes and that does not require a huge number of weights and physical interconnections. . Moreover, the topology of this network must not only be realistic and achievable with a limited number of hardware, but also be capable of operating at very high speeds. The neural network system disclosed herein meets these demands.

[0008]

To achieve the above object, the invention as claimed in claim 1 provides a feedforward segmented neural network comprising a number of network layers and a plurality of sub-networks. Each subnetwork is deployed in one of a number of network layers, at least one of which contains a number of subnetworks. All sub-networks in a network layer with multiple sub-networks are separated or separated from other sub-networks in that network layer. The multiple network layers are arranged in a pyramid shape from one input network layer to one output network layer such that the number of sub-networks in the network layer decreases from the input network layer to the output network layer. . In actual hardware, a storage device is included in the secondary network.

In order to achieve the above object, the invention according to claim 2 provides a method for constructing a fast feedforward segmented neural network. This method includes the following steps. That is, the step of selecting a sub-network model and the step of applying the selected sub-network model by the segmented neural network (the segmented neural network has a large number of network layers in which a large number of sub-networks are arranged. , And the step of training the selected segmented neural network, while imposing the constraint that the values given by one network layer to the adjacent network layer include binary signals, and The steps of mapping the input and output values into a segmented neural network model and programming the inputs and outputs of the mapped sub-network in a storage device.

[0010]

The present invention provides a neural network system which uses a large number of inputs and nodes and which, in its implementation, adapts itself to realizable hardware to enable fast pattern recognition. This system, referred to herein as a "partitioned" neural network system, includes multiple network layers stacked in an upward-facing pyramid shape, the network layers being composed of multiple "sub-networks." Within each sub-network there are a plurality of nodes, which are arranged in fully interconnected layers and / or partially interconnected layered arrays. Each sub-network includes a plurality of inputs, and in the embodiment described below, 1/1 / the number of inputs
It has two outputs. Both the input and output of this sub-network are 1-bit digital values limited to 0 or 1. Any number of nodes in any number of layers can be modeled within each sub-network.
Furthermore, the number of outputs of the sub-network may be other than 1/2 of the number of inputs.

The first network layer is the input network layer and the last network layer is the output network layer. All other intermediate network layers are latent network layers. Each network layer can be modeled for any number of sub-networks, where each sub-network is independent of all other sub-networks in that network layer.
In this way, each network layer is segmented.

For a given number N of inputs per sub-network and a number M of sub-networks within a given network, this network layer has a number N × M of inputs, and in the embodiment described below the number of outputs is ( N × M) / 2.
The output from each network layer is such that the output of the preselected subnetwork in the previous network layer is connected to one subnetwork in the next layer. Will be input. The size of each subsequent network layer is determined by the number of inputs required for the network layer below it, the next network layer in the pyramid. The number of inputs in each subsequent network layer,
The number of sub-network layers and the number of outputs are reduced (ie 1/2) from those of the previous layers. The network layer is stacked on the previous network layer until there is (for example) only one sub-network in the top network layer. Thus, a pyramidal, segmented neural network is constructed.

By forming a sub-network with a finite, technically possible number of inputs and outputs, the sub-network can be replaced by a simple storage device, for example a look-up table in RAM or PROM. After training the network by computer simulation, the input and output values of each sub-network can be mapped and stored in storage. Furthermore, each sub-network can be replaced by its corresponding storage device. Once the sub-networks are replaced with hardware storage, they can be used to the extent comparable to neural networks in forward mode. Although the speed of this system is high, the speed is generally determined by the access time of the storage device used and the number of network layers of this system. Furthermore,
The number of physical interconnections is relatively small.

[0014]

DETAILED DESCRIPTION OF THE INVENTION A more detailed method of the present invention will now be described with reference to the drawings, wherein the same reference numerals are used in different figures to designate the same or similar components.

One embodiment of a neural network system is shown generally at 10 and, in accordance with the present invention, FIG.
Is illustrated in. Neural network system 1
0 is a pyramid-shaped section with five network layers called "multilayer 1" (ie input layer), "multilayer 2", "multilayer 3", "multilayer 4" and "multilayer 5" (ie output layer). It is a generalized neural network. Except for "Multilayer 5",
Each layer of neural network 10 includes a number of sub-networks "SN" 12, each of which "sub-network".
Is assumed to consist of 16 inputs and 8 outputs. Each "sub-network" consists of a storage device such as a PROM, and the number of outputs of each sub-network is 1/2 of the number of inputs.
Is. For example, the output of a pair of sub-networks in "multi-layer 1" is given to the same sub-network of "multi-layer 2", while the output of a pair of sub-networks of "multi-layer 2" is given to the same sub-network of "multi-layer 3". Given, the outputs of a pair of "multilayer 3" sub-networks are provided to the same sub-network of "multi-layer 4", and the output of "multi-layer 4" sub-network 12 of a single sub-network of "multi-layer 5". It becomes an input (as will be described later, the pairing of the output of the sub-network 12 is predetermined in the system 10).

The network 10 is used only in the forward mode, the training is carried out on individual computers and the training results are programmed in the sub-network, more specifically in the storage used. Thus,
In practice, the entire network consists of a number of sub-networks, each sub-network for example consisting of 16 inputs and 8 outputs. The eight outputs of each sub-network of the first layer are combined into 16 bits at a time to be the inputs of the sub-network of the second layer. This construction is repeated until the last layer, where the eight outputs are the result of the whole network.

The neural network system 10 is
Input information for "Multilayer 1" is received through a plurality of input latches 14 disposed within a computer processing system, such as a pattern recognition system, and connected to receive data directly from a computer data bus 16. As an example, 32 8-bit input latches (type 74ALS
273) can be applied to the 256-input neural network. The information from the pre-paired latches is sent asynchronously to the corresponding sub-network in "Multilayer 1". Input latch 14 is loaded synchronously via the common data bus by activating the appropriate valid lines EN [0 ... 31]. Each sub-network of the pyramidal neural network structure again has an AM27512 with 16 inputs and 8 outputs.
It can be implemented as a 64k PROM. The output of the neural network (ie, "multilayer 5") is buffered by the output buffer 18 and finally the computer data bus 16
Given to. The buffer 18 is an integrated circuit type 74ALS
244.

As yet another example, an embodiment of the sub-network 12 according to the present invention is shown in FIG. In this example, three node layers are shown. Input node layer 20 includes a plurality of nodes 22, each of which receives 16 inputs to subnetwork 12. The output of the input node layer 20 is given to the intermediate node layer 24, and the output of the latter is inputted to the output node layer 26,
Becomes the output of the sub-network 12. This embodiment of the sub-network constitutes a fully interconnected sub-network, where each node in each layer receives the same input and each node outputs to each node in the next node layer. Because.

As shown in FIG. 3, each internal node 22 constitutes one non-linear addition means. The node multiplies each input (x) by the respective weights (w1, w2, ... Wn) and sums them, and then performs a non-linear transformation using the sum. This non-linear transformation has a characteristic that it is asymptotically bounded at both positive and negative infinity positions, and further has a characteristic that it monotonically increases. There are some normal functions with the above properties,
One of them is the Tanh (x) function. Those skilled in the art will appreciate that there may be other functions with the above properties.

As first mentioned with reference to FIG. 1, the network is made up of layers of storage devices, each storage device having 16 inputs and 8 outputs. As an example, FIG. 4 shows pin connections for an AM27512 type PROM device. Again, other storage devices can be used. Each storage device can be regarded as one sub-network, which (in the embodiment described here) corresponds to 16 input and 8 output nodes as shown in FIG.

Construction of a neural network according to the present invention will now be described with reference to FIG.
During network training, the network is modeled as a segmented network, but its inputs are, for example, 16 simultaneously received and they are passed to the corresponding individual sub-networks. That is, reference numeral 30 is “modeling of a feedforward network as a segmented network”. This sub-network is modeled as a fully interconnected multi-layer network with 16 inputs and 8 outputs. Although any commercially available computer simulation technique can be used for network training, it is only subject to certain limitations. That is, reference numeral 32 is “training the network while imposing restrictions using computer simulation”. To be clear, the training is done so that the value sent from one network layer to the next is a binary value, 0 or 1. Since this condition does not apply inside the sub-network, the internal state of the sub-network can be a continuous real value during the training period.

After network training is complete, a full input / output mapping is done on each sub-network. That is, reference numeral 34 is “creation of complete input / output map of each sub-network”. Once the complete input / output map of the sub-network is obtained, the internal states are no longer needed and the input / output relationships can be directly programmed in the selected storage device. That is, reference numeral 3
No. 6, “Programming of input / output relationship in storage device”.

In the case of the network shown in this specification, 2
There are 56 binary inputs (see Figure 1). This binary input is latched from the computer interface, one byte at a time. These 256 inputs are then input to FIG.
It is sent to the storage device of the 16-input layer sub-network, labeled "Multilayer 1" therein. Each 16-input sub-network store produces 8 outputs to send to the next network layer. Their outputs are sent asynchronously to the next network layer as inputs. Therefore, each successive network layer contains as many storage devices as the half of the previous layer, and as a result, "multilayer 2" in FIG.
The second layer, labeled, must have eight storage devices. The same goes on until the last layer of the pyramid structure, the "multilayer 5", has one storage device. Its output layer has 8 outputs, which makes the network 256
It becomes possible to make street decisions or classify.

Once all 256 inputs are latched to the network inputs, the network output is delayed by the propagation delay of the storage device from one layer to the next. For example, in the case of the storage device shown in FIG. 4, the input / output delay time is 30 ns. Therefore, in the case of 5 layers, the input / output delay time of the entire network is about 150 ns.

Each sub-network has 11 potential in-layer nodes, which makes sense as a fully interconnected network with 16 inputs, 3 layers, and 8 outputs.
The number of interconnections of each sub network model is 266.
Since the entire network consists of 31 sub-networks, the total number of interconnections between sub-networks is 240
become. Therefore, the total number of equivalent interconnections is 31 × 264.
It becomes + 240 = 8,184. This is 54 per second
This is equivalent to the astonishing speed of 0.0 billion interconnections, and at least the speed that can be achieved by the conventional neural network configuration is at least 1.
Equal to 0 times.

The invention also comprises the following examples not explicitly listed in claims 1 and 2. The feedforward segmented neural network of claim 1, wherein: (a) each of the plurality of sub-networks is implemented in hardware as a storage device.

(B) The feedforward segmented neural network of paragraph (a) wherein each of the storage devices is a RAM device or a PROM device. (C) Each of the storage devices has M inputs and N outputs,
The feedforward segmented neural network of claim 1, wherein M> N. (D) A feedforward segmented neural network having the characteristics of the terms (c), wherein the number of outputs N of each of the storage devices is half the number of inputs M, that is, N = M / 2. The neural network, wherein the size of the network layer is half the size of the preceding neural network layer. (E) A feedforward segmented neural network having the characteristics of item (d), wherein the storage device has 16 inputs and 8 outputs. (F) The feed-forward segmented neural network of claim 1, wherein each of said sub-networks is a fully interconnected or partially interconnected neural network. (G) A feedforward segmented neural network having the characteristics of (f), each of the sub-networks having a plurality of sub-layers, and each of the sub-layers having a plurality of processing nodes. network. (H) The construction method according to claim 2, wherein the programming step (d)) inputs / outputs the sub-network mapped in the step (c) for each sub-network.
A method comprising programming an output in a storage device. (I) A construction method having the characteristics of (h), wherein the segmented neural network modeled in step (a) includes a neural network model having a large number of network layers, each network layer being an even number. A network having a number of said sub-networks. (J) A construction method having the characteristics of the term (i), wherein the selection step (a) further selects a segmented neural network model, that is, the number of inputs, the number of sub-networks, and the output of each network layer of the model. A method including model selection such that each of the total numbers is half the total number of inputs, sub-networks, and outputs of adjacent network layers. (K) A construction method having the characteristics of the terms (g), wherein the training step (b) includes training the segmented neural network selected in step (a) using computer simulation.

[0028]

As described above, according to the neural network and the method of constructing the neural network of the present invention, it is possible to deal with a speed which cannot be dealt with by the conventional techniques related to today's neural network, and the design is simple. Is never lost. This is accomplished by not fully interconnecting networks, although the term "partitioning" of networks is used herein.
In addition, by not connecting the networks completely, this design can be extended to neural networks with a large number of inputs, which is practically unrealizable with hardware as a fully interconnected neural network. The speed of this neural network is
Limited only by the number of network layers modeled and the speed of the storage used.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of a segmented neural network according to an embodiment of the present invention.

2 is an illustration of a sub-network for use in the segmented neural network of FIG.

3 is an explanatory diagram of one node in the sub network of FIG. 2. FIG.

4 is an illustration of a PROM sub-network according to the present invention for the neural network of FIG.

FIG. 5 is a flow chart of a neural network construction technique according to the present invention.

[Explanation of symbols]

 10 Neural Network System 12 Sub-Network 14 Input Latch 16 Computer Data Bus 18 Output Buffer 20 Input Node Layer 22 Node 24 Intermediate Node Layer 26 Output Node Layer

Claims (2)

[Claims] 1. A feed-forward segmented neural network, comprising: a plurality of network layers and a plurality of sub-networks, each sub-network being contained in one of the plurality of network layers; At least one of which comprises a plurality of sub-networks, all of said network layers comprise at least one sub-network, and all sub-networks within each of said network layers are A plurality of sub-networks separated from the sub-networks, the plurality of network layers are configured in a pyramid form from the input network layer to the output network layer, and the number of sub-networks in the plurality of network layers is the From the input network layer to the output network layer A neural network characterized by a decrease. 2. A method for constructing a feedforward segmented neural network, comprising the steps of: (a) selecting a subnetwork model and a segmented neural network model using this subnetwork. The model of the network is a model having a large number of network layers in which a large number of sub-networks are arranged, and (b) training the segmented neural network selected in the above (a) step, while Requesting that the value sent from one layer to an adjacent network layer includes a binary binary signal; and (c) completing the step (b) one after another, and each sub-component inside the model of the segmented neural network. Map network input and output values And (d) for each sub-network, the step (c) of programming the input / output of the sub-network mapped in step (c) in a binary device. How to build.