He isn't having a relationship with his brother but with someone else and just tells him that on the past he was in love with his brother but no more
Boku no Kareshi no Sukinahito