So if one wants to say, I am loved, one may want to say ego sum amato. It would make sense for the past participle to agree with the subject.
Some may think it unnecessarily complicated to have an actual passive conjugation. I don't think it's a problem - it could be based on a simple transformation from the active conjugation scheme. This is what I'm thinking:
For the verb to be loved, the infinitive will be amari, and the six conjugations will be amor, amar, amatur, amamur, amatir, amantur.
Thus, the passive conjugation is easily identifiable and formed by having a final -r (adding -r if the active form ends in a vowel, -ur if it ends in -t, or changing its final -s to a -r).