Physics Informed Neural Network — A neural network that understand your physics (Part 1)
Neural Network are often called as the universal approximator. Given enough data and enough depth in their network, it was able to approximate any functions. You just have to find the right data, and design the right architecture. But in practice, this is hardly the case. Your data needs to be good enough, your network needs to be comprehensive, and when you expand your network large, you simply did not have enough computational power to train and run the network.
That makes neural network not favourable to problems already solved effectively by first-principle modelling. Something like thermodynamic control in petrochemical plants, orbital mechanics of satellites, and even numerical weather predictions. This problems doesn’t need high amounts of data, and high computational power to run, and it is quite effective currently.
But, it doesn’t mean that we cannot use neural network at all to model this problems. We just need to find the right framing. How about instead of just relying on the nueral network, we add that first-principe modelling inside the neural nets. With that, we can get best of both worlds. We can get the potential of universial approximation, while at the same time ensuring that our network doesn’t stray far from the phsyical laws.
So let me introduce you to PINN — Pyhsics informed neural network
The beauty of PINN
The idea behind PINN is quite simple actually. We know that our first-principle modelling is not always accurate if we want to compare to real world conditions. For the purpose of explaining, let me give you an example. But in order to do that, we have to bring you to the world of chemical engineering, and introduce you to chaotic world of Thermodynamics.
No, don’t be afraid. This is just only a shred of thermodynamics. It’s about heat and energy. In a petrochemical plant, we have an equipment called furnace. Which I think you might have heard of. It’s function is to heat up a medium/process fluid. The medium will be transferred via piping to the furnace, and the piping will be heat up via combustion inside the furnace box. So if you remember your school science, you would know that the heat is transferred via radiation, conduction, and also a little bit of convection.
But the main provider of that heat, is coming from that combustion, which is formed through chemical reaction between the fuel (in this case methane gas), oxygen and heat (remember the fire triangle).
In thermodnamics, we can model the entire furnace system with this simple equation:
where the incremental temperature increase is equal to the amount of of the transferred from the combustion minus the heat kept by the ouflow medium (process fluid),
However, in reality, we cannot just simply assume the heat that we transferred from the combustion will all end up in the outflow fluid. This is just a simplified representation. We must account for loss. There’s heat loss to the furnace wall, there’s heat loss during combustion itself, and there’s also heat loss during the transfer itself.
So all of that loss is hard to model correctly and has been quite a challenge for chemical engineer to implement. Therefore, most of them can get the best approximation only to the reality inside the furnace.
But, what if, we model the loss part to the neural network. And use the power of universal approximation to model the loss perfectly?
And that is what we’re going to do with PINN.
No Architecture, just Loss
The key to PINN is just the loss. We know the loss is the guide to how we update our network parameter weights. That loss is taken from the loss function which is different for different type of problems/network. But typically, it is the difference between ground truth and predicted value from the network. The loss is calculated during training, and it will guide how the network is updated.
So, for PINN, we will add additional loss on top of the typical loss that we find during training. This loss would be the pyhsics loss. So let’s model that using our Thermodynamic equation that we have established earlier.
In order to do that, we must model our furnace thermodynamic equation as a continous time problem. Means, we have to change it into derivative using ordinary differential equation as below:
therefore, the physics loss can be model as below:
During training, we will feed some of our features value inside this loss functions. However, we have to keep some of it modeled as trainable parameters, that can be updated during training. For this case, to keep things simple, we will just model the constants (the Cp) as the trainable parameters. This is also the same case for the heat. As some part of heat are also held up by constants. That will also be modeled as trainable parameters during training.
But, if you want to make your network as accurate as possible, you can always calulate the constants by providing more empirical data. That is the beauty of PINN. More data leads to more accurate first principles, and more accurate network
That will be our physics loss. So, on top of our data loss,
Which take the difference between actual and predicted, we will have the total loss which combines both, and we can write it as below:
This total loss will help us in guiding our neural network to understand the phsyics and hopefully, some of the nuance of that thermodyanamics will be embedded in the parameters.
Before we ends
So, the next part is just the implementation part in our code. We will provide real data that I have taken from a petrochemical plant, and feed that to our training process. So we will pick this up hopefully in the next part. See you guys then…