Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why can't you model each neuron as an individual object with a unique UID in code?

I am fortunate enough to have a friend that's fairly accomplished in ML and a really nice guy. He spent a video chat + code session with me where he helped me build an NN from scratch in Typescript, we replicated the network in the paper here:

https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...

Here's the Codesandbox link, if you notice the console output it (approximately) matches the figures from the paper above:

https://codesandbox.io/s/node-typescript-vqszi?file=/example...

I don't really understand much about ML or neural networks, but from a code perspective, if you have these data structures:

    class Neuron {
      id: number | void = neuronUidGen.next().value;
      incoming: Connection[];
      outgoing: Connection[];
      rate: number;
      bias: number;
      output: number;
      error: number;
    }

    class Connection {
      id: number | void = connectionUidGen.next().value;
      from: Neuron;
      to: Neuron;
      weight: number;
   }
Why couldn't you manipulate these however you see fit? There's a Layer implementation in there that's commented out because it doesn't function right that could roughly accomplish something like this theoretically, right?


You could. But that's not the hard part. The hard part, for the foreseeable future, is that if you write your code like that, you will find it very difficult to get decent performance out of existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. (Oversimplifying a bit.)


But the Neuron and Connections are just metadata objects like pointers that hold arrays, I don't understand.

Surely in PyTorch/TF, the ND-Arrays of NN's must also be associated to an object reference?

> existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape.

    const dot = (inputs: number[], weights: number[]) => {
      let total = 0;
      const length = Math.min(inputs.length, weights.length);
      for (let i = 0; i < length; i++) {
        total += inputs[i] * weights[i];
      }
      return total;
    };

  activate(input?: number): number {
    if (!this.incoming.length) {
      this.output = input;
    } else {
      const inputs = this.incoming.map(connection => connection.from.output)
      const weights = this.incoming.map(connection => connection.weight);
      this.output = squash(dot(inputs, weights) + this.bias);
    }
    return this.output;
  }
This is on principle the same concept right?


Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... (I oversimplified things in my comment above).


Could there be a transpiling step that converts the objects to arrays, targeting PyTorch or TensorFlow?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: