Well the word "aggression" can technically include nonphysical actions. Psychologists say that women manifest their aggression in the form of gossip and reputation destruction. None of that is physical.
However, that word does NOT include unintentional actions. If someone physically harms you on accident, it is technically incorrect to say that that person committed an act of aggression.

So it seems like your'e considering the human being as a whole; you aren't differentiating between impact and intent because intent is just another form of impact (caused by biology and evolution). In other words, you're implying that we actually have no voluntary intentions. Is that correct?
If it is, it basically sounds like you're arguing that there is no such thing as free will. That's perfectly fine but I'd argue that that argument isn't relevant to the current topic. What I mean to say is that, even if there is no such thing as free will, there is still such a thing as intent; it's just entirely caused/determined by causal events in nature. whether or not we have free will, we shouldn't put any more or less conviction into our behavior. If we decide that there is no such thing as free will, and that theory turns out to be correct, it's not like we can just decide to stop thinking and stop trying and our bodies will just continue to move on their own without us actively trying to make them move. In other words, the causality in nature, whose existence disproves the existence of free will, determines our intentions. It isn't that intentions don'e exist; rather, it's that our intentions aren't under our control. Sam Harris once said something like "we do make choices but we have no choice as to which choices we make."
Of course, we can't know someone's intentions for sure. However, for many reasons, we can get a pretty good idea if something is an accident. If we bump into each other on the sidewalk, you can't know that I didn't walk into you purposefully but you can probably accurately infer that if I say something like "whoops, sorry." Similarly, if I unintentionally offend someone but apologize afterword, and if the apology would seem sincere to the average person, I believe that we should be able to forget it.

@BFrydell Taking offense and apologizing - these are binding part of social contract that we establish. This social contract varies across the world. In some cultures, people don't take offense for bumping into each other maybe and in some cultures apologizing for bumping into each other might not be a norm. What I am trying to say is that, we cannot come up with an absolute law based on intentions because intentions are not in one's control (as you said it yourself).

As for putting more or less conviction into our behaviour, I think the desire to be convicted about one's behaviour is also an intent in itself over which one has no control. It's paradoxical in some sense. The thought to stop oneself from doing something is often cited as an example of disproving free will. However, the thought that arose within oneself to not do something is also part of the uncontrolled cause/effect mechanism that is playing out. If free will doesn't exist, it includes all forms of intent - be it an intent to do something or an intent to not do something.

The only thing we can do in order to minimize the impact of bad behaviour is continue to set examples of good behaviour and educate our children about good behaviour and HOPE to see a reflection of that good behaviour in the society. However, most of our education in schools at the moment doesn't emphasize on behaving well or thinking well but emphasizes more on math and science. So what if someone behaves badly? - intentionally or not doesn't matter. What matters is there is a bad behaviour or impact. How do we make sure a similar impact doesn't occur in the future? Only by emphasizing the value of good behaviour. I think it all boils down to good wholesome education and care from the very young age.

Well I wasn't really talking about the court of law. I was more curious about the court of social relations.
If I were ask someone from India teach me a phrase in his/her "native language," that could technically count as a microaggression. I could not be taken court for that but there might still be a conflict (depending on his/her temperament). resolve that possible conflict, should we more attention my inquisitive and innocent intentions or should we disregard my intentions and more attention the impact (which, in this case, would be the offense that he/she has felt)?

What details, do you think, are more important than intent?